#200 - ChatGPT Roadmap, Musk OpenAI Bid, Model Tampering
01:48:09 Share

Hello and welcome to the Last Week in AI podcast where you can hear us chat about what's going on with AI. As usual in this episode, we will summarize and discuss some of last week's most interesting AI news. And you can also go to lastweekin.ai for all the links for this episode and also for our text newsletter. I am one of your hosts, Andrey Karenkov. My background is of having studied AI in grad school and now working at an AI startup.

And hey, everybody, what's up? My name is Jeremy Harris. I'm your other co-host, co-founder of Gladstone AI, an AI national security company. You know that if you're listening to the podcast every week. So I must get tiring to get the bios every time, but at least now you know we're, I guess, you actually know who we are. I wonder, we'll maybe consider retiring your bios every time. I don't know how many new listeners we get that justify that. And I guess we can go in and jump to

addressing the fact that this will be the 200th episode of the Last Week in AI. There's a few more on the podcast feed due to some interviews and stuff, but this will be number 200. Damn, I feel old. I know. Let's quickly celebrate that for a bit. Amusingly, there was also an Apple podcast review that just said, don't ever stop.

So I guess so far, pretty good. 200. We've had a good record of doing this. Jeremy, you joined at episode 110 for reference. Okay, so I'm still, yeah, I'm on episode 90. Yeah, you're on episode 90. For anyone who hasn't been a long time listener, we were doing this back starting 2020. We had a different co-host. Thank you.

Podcast almost did stop, actually, in 2022. We had a bit of a break, and then thanks to Jeremy, we were able to come back and keep going. So we'll see. Hopefully, we won't stop at least until AGI gets here. I think that's pretty good.

That gives us at least two weeks or so. I know. Given that, maybe a good time to make some changes. Maybe we'll retire the bios starting next episode. And we'll also start discussing listener requested topics towards the end of a show. Since we have gotten more of those kinds of things, that'll be kind of the last section after all the news. The only thing we'll

be talking about as far as listeners, is any corrections that we need to address. So maybe that will be...

better for a lot of listeners. Quick preview of what we'll be talking about in this episode. We have some cool new models in tools and apps. Adobe is coming out with a new SOAR arrival and we have some updates to kind of the timeline of LLMs. Applications and business, of course, more opening eye drama with Elon Musk. Got to talk about that. And some kind of previews of what's coming with other LLMs like Anthropic and

Got a few neat projects and open source

stories with one text-to-speech model and some data sets, research and advancements. Similarly, talking about some distillation scaling laws and some tampering attacks for LLMs, which would be cool. Policy and safety, a bit of a mix of discussions between the US and UK, some geopolitical stuff, and paper also on utility engineering.

And we'll cap it off with a discussion of some updates to AI copyright law, which has actually happened at long last. Yeah. And then we will cap it off with a bit of discussion of listener requested topics. So hopefully you won't go forever and we'll actually get to that before we are completely out of energy.

And with that being said, let us get going. In tools and apps, our first story is Adobe's Sora Rivaling AI Video Generator is now available for everyone. So Adobe has had a whole suite of generative models of their own, starting with Firefly. Now their video generator that they call Generate Video is out for public beta access.

And it is available through the Firefly web app, which also has image generation and some translation stuff.

There are two ways you can use it, text-to-video and image-to-video. So pretty similar to other things, videos are output in HD, 1080p, 24 frames per second, and you can generate them in 90 seconds or more. So it takes a minute and a half, two minutes to generate something. And the length of those is up to five seconds.

So yeah, this is a preview, of course, and I guess we'll see how long it takes to actually go out into their mainline tools. They have been integrating various generative models into Photoshop, for instance, with generative fill.

And Adobe is marketing this Generate Video tool as a production-ready kind of tool for use in films without any copyright issues, which has also been their kind of line with all of their generative models that this is something you can use when making a film and not worry about people suing you. So interesting to see if this actually does have a leg up against Sora and some other competitors.

Yeah, it really feels like we're reliving that kind of on-ramp, the generative image, like text-to-image on-ramp that we felt two years ago, two, two and a half years ago. Over the course of about 18 months or so, you went from

Pretty slow, frustrating, unremarkable capabilities to like, you know, photorealistic stuff. And now from one release to the next, you can't really tell the difference. I will keep beating this drum. I think this is kind of a race to the bottom on pricing. Ultimately, it's going to happen here as well, where everybody's just going to be charging essentially for the cost of inference plus some small margin.

This is really looking like it's going to be commoditized as well. I don't really see an intrinsic reason why video would be any different from images. But just to give you a sense still of the roadmap. So the video output that they can provide here is in 1080p, 24 frames per second. So decent-ish, maybe a little on the low end, but it is an upgrade from the original 720p quality. So we're getting more and more into deeper HD here. Interestingly, still choosing to stick with that five seconds, right? By contrast,

Sora can do 20 seconds already. So you sort of see this trade off, like where do you put the compute? Do you put the compute in generating longer videos that are logically coherent across longer periods of time? Or do you invest your compute in higher resolution, your 4K, your 1080p, that sort of thing?

By the way, Adobe does say they're also working on a faster, lower resolution, what they call an ideation model and a 4K model, which are both coming soon, supposedly. But this is, I guess, an interesting use case and a bit of a bifurcation that at

I'm sure there are things like this in text to image. You're going to have to excuse me because I don't know that space quite as well other than what we talk about on the podcast. In every domain, we get deep enough, there are interesting trade-offs. With video, it's extra obvious, this idea of do I go higher resolution? Do I put my computer into higher resolution frames or into coherence across a larger number of frames?

You can see why OpenAI might have gone for the coherent side more than Adobe with these 20-second videos, just because coherence is at least intuitively a bit closer to the sort of reasoning that you need to make AGI. You have to be able to track objects and their interactions across longer periods of time. That starts to look more world model-like than necessarily a high-resolution image.

Again, you know, both there's some truth to both of them. I just intuitively think maybe the longer time resolution of opening eyes Sora is a little bit more in tune with that. So maybe an interesting niche for them to carve out for themselves here at Adobe focusing on, you know, higher resolution video quality, 4k and all that.

Right. And I think we did discuss this preview. They posted some videos of it prior to which will be available. So they are trying to differentiate in a couple other ways. They do say that, for instance, you can

prompt to model with camera controls as one of the kind of common things, things like zoom out and give me this particular shot. And on my end, I will keep beating the drum, but I think image to video is a much bigger deal than text to video in general, as kind of an application of special effects of

you know, removing weird artifacts in your video, all sorts of practical applications I can see being used. So wouldn't be surprised if they're also emphasizing that.

And also worth covering, in addition to this coming out, they have introduced new subscription tiers for Firefly with credits for these models. So there's a Firefly standard subscription, which costs $10 per month, gives you 200 video or audio credits.

Not too sure how much that converts to. Seems like maybe 25 second video generations. So not a ton. And there's the pro subscription, which is $30 a month, which is 7,000 credits. So about 75 second videos. So starting to commercialize it. I wonder how this will connect to their existing subscription tiers for their tool use for things like the editing suite. Next story.

And moving on, next we have OpenAI and them saying what will get to GP5 as opposed to anything that they are releasing. So Sam Altman posted on Twitter slash X where he had not a super detailed but a fairly specific idea of a roadmap they're looking at. And

The highlight is that they are trying to move away from the current paradigm where you have like six models to choose from. They have GPT-4.0, you have all one preview, I think still, or free mini now. And if you use ChatGPT, you have this model drop down where you need to

select which model you want to use for any given task. And you have various amounts of token usage per model. So it all gets a little bit cumbersome as a specifically new user as well. So what they're saying is that the new series of models

we'll be sort of unifying this O series of reasoning models and the GPT series of models, where the model will just sort of know what to do based on your prompt. And so we won't be seeing

Presumably, O3, we won't be seeing anything until we get this kind of unified tool. And apparently, when it does come out, free ChatGPT users will have unlimited access at what you're calling the standard intelligence level. And plus subscribers will have a higher intelligence level. So...

Yeah, it seems like they're going to change up for user experience, which is interesting given that that hasn't really happened since the early days of chatbots. Yeah, it's been good inspiration for me too. Anytime I offer any consulting work, I now have the two tiers. I offer them the standard intelligence level and then the high intelligence level.

That's a good business model. It really is. This podcast, if you're wondering, is high. This is as high as I can go. So don't bother. But anyway, so one interesting update too is

We didn't know that this was going to happen, right? The big update here is the bundling, essentially. As Sam says somewhere in his tweet, right? He's like, we hate the model picker just as much as you do and want to return to magic unified intelligence, right? They want this feeling to be that you show up at the interface at the console or whatever, you pump in your query. You just kind of get the answer from the model that's most appropriate for your query. And that does make sense. We did get, I think a couple of weeks ago, Kevin Weil, OpeningEye's CPO,

did say that they were on track to release O3 in the kind of February, March timeframe. So this is still consistent if you kind of include it in that roll up. Though Sam maybe gave himself a bit of wiggle room. He gave this ambiguous timeline of when these things would roll out saying it would be weeks slash months for GPT-5 at least.

So there is a little bit of ambiguity, but as you said, I like what you said, not high detail, but high specificity. That does seem like we're getting here. We're finally getting named releases.

One thing to keep in mind, right? So GPT-4.5, Sam Altman is now saying that is going to be the model internally that was codenamed Orion. We saw a big stink about that for a long time, right? The Strawberry Project, which turned into O1, and then Orion, which is turning into GPT-4.5. And we've had this conversation on a podcast before. A lot of people talk about, you know, what will GPT-5 be able to do? What will GPT-4.5 be able to do?

And it's really kind of, in a sense, pointless to talk about things in quite that way. Some people use that colloquially to mean like really the next order of magnitude or two ooms of scale, which makes total sense. But for others, there's this kind of fixation on just like the incrementing of the name of

And that's just something to keep in mind. OpenAI internally is building a crap ton of different models. They don't themselves know which one they're going to call what. And so for a long time, this thing was just called Orion, and that was just the model under development. And if it seemed like the vibes were right, they would release it as GPT 4.5, which happened here. But that wasn't locked in initially. So just...

Just to kind of clarify there, if you're looking at speculation on GPT 4.5, whatever, the ooms are the thing that matter a lot more. The actual orders of magnitude of compute that's thrown at these models, the names, not so much. But in any case-

At least now we have a name to hang on to what was previously the sort of secretive Orion base model project. And by the way, that is a base model. So GPT 4.5 is going to be the last non-chain of thought model that opening eye ships. From now on, apparently, they're all going to be focused on, well, chain of thought is a loosely defined thing, but they're going to be focused on models that use the kind of 0301 reasoning paradigm. That's at least going to be the default paradigm.

And they will still have these models like GPT-4.5 that can one-shot, but the goal is to kind of orient more and more towards reasoning. Right. And to preview a bit, we will be talking about some similar sounding plans, Anthropic, down in the business section because it was some business use, but...

The idea of unifying reasoning models of non-reasoning models seems like it's kind of up in the air where there's no clear need to separate, right? You can just enable a model to be able to reason if it wants or thinks it's warranted or not to reason and just to output without this long trace.

which actually does seem like a bit of a no-brainer when you think about it, like why do you need to separate these out? And it seems like a future maybe that you don't, that you have a single model capable of both paradigms. And as to the naming, I do think it's a bit interesting to just take a quick look back

ChatGPT when it first came out late 2022 was built on top of GPT 3.5 at the time. GPT 4 came out in March of 2023. And then since then, we've gotten into the O model family, which was Omni because they added things like audio to it.

GPT-4.0 and now they have O1 as their reasoning model. So they did basically stop incrementing in favor of having new types of model names to reflect new capabilities and new types of model from GPT-4, where it used to be GPT-2, 3, 3.5, 4. As you said, that was always just going up in scale and improving on

on your intelligence levels. And since then, since mid-2023, we've seen kind of more focus on broadening what the models are capable of in terms of inputs, outputs, and I guess the way they do reasoning. Next story.

Next up, we have another bit of news from OpenAI. Apparently, they're rethinking how AI models will handle controversial topics. So this is a part of a released 60-page model spec, which is their guidelines for AI model behavior. And this updated spec does include details as to how to handle these controversial topics.

And seemingly, they will make it so the AI is able to do some slightly more, I don't know, spicy things. So for instance, there would be the ability to allow certain adult content in appropriate context. And there may also be the ability for users to kind of enable or disable safety guardrails to customize that a bit. So yeah,

Yeah, again, not perhaps too surprising. I think many people using LLMs do complain or criticize when models are overly restricted and overly careful. So them moving away a bit could be a reasonable move. Yeah, it's...

The model spec also is something we've covered before, but just if you're new to the concept, in some ways, it sounds like a document that might be fine-tuned on, maybe just thrown at an LLM and fine-tuned on this, and that tweaks its behavior.

Some truth to that, but mostly it's a meta instruction document. It's not just something that you load into a model's context or just use as fine tuning, but generally describes the approach OpenAI takes and that it recommends developers take at every level of the stack from data collection to training, to serving the model, to using it in applications. And so it's sort of like their overall meta instructions to themselves and to others,

It is a lot more detailed than it was previously. And as you said, there's a lot of emphasis on what they call intellectual freedom here. Sort of interesting.

One might be tempted to suspect that this is sort of a response to the change in administration, that previously they were perfectly happy, Sam Altman having been kind of a longtime Democrat, they were perfectly happy to kind of align with the sort of more, I guess, the more kind of woke orientation of the chatbot as it had been previously. Now kind of switching over, which is a very Sam Altman thing to do, kind of go with the winds of change. And

And that's sort of what you're getting here. It is useful. And I think a lot of people, as you said, they're getting frustrated with dealing with the model that was kind of overly constrained previously. So yeah, I mean, there are a whole bunch of ways in which this thing gets used. It's almost like pointless to go through them all. But one sort of sub aspect here is there's like this chain of command that's outlined in the model spec that

describes how the models are meant to prioritize different levels of instruction. You can think of their platform level rules that OpenAI uses to override everything else. These are like safety safeguards or legal compliance things. But then you've got a system prompt. So how do you rank the system prompt and importance relative to the prompts that the developer puts in? And they've got all kinds of measures, and we covered a paper that deals with this as well.

It also contains instructions that they give to human labelers during reinforcement learning for human feedback data collection. And so the model spec really is a very all-encompassing document in that sense. And yeah, there's a big blog post that comes along with it where they're sort of celebrating the higher adherence of the O1 series relative to the GPT-40 series.

to their spec. So, so they see, you know, across all these things that they care about, they have these funny sounding sort of category names for the, for the, the things they're trying to quantify and care about. So adherence to the chain of command that we just talked about staying in bounds. So essentially giving responses that, that are kind of that are,

consistent with the prompts they've been given. Seeking the truth together is another one, sort of interesting. Apparently a big improvement there of 01 relative to 40. Do the best work. So some notion of optimization for the actual correctness of the answer, use appropriate style. So on all those, you see improvements. You actually see

A worsening of performance though on be approachable. So 01, apparently less approachable than 40. That at least tracks my experience. But anyway, kind of interesting and you sort of see it all laid out there. So they're letting us know as well, they won't be publishing more blog posts with every update to the model spec, but they do have this log that you can track, which is great for transparency, right? You go check it, see how they've updated their meta instructions. Next story.

And next we have Perplexity AI announcing a new ultra-fast model. So they have this new version of their own in-house model, Sonar, which is built on top of Lama 3.3 7B. And they're basically saying that this is much better than iModels.

And I have other off-the-shelf models like Cloud 3.5, Haiku, or GP4.0 Mini for the particular thing that Perplexity does, which is AI-enabled web search. It's one of the ways similar to GPT web search, to Gemini web search. You enter a query, it looks up some websites, and the AI then reads those websites and either answers your question or summarizes or whatever.

So this one, Sonar, is fine-tuned for improved factual accuracy and readability. And they are claiming that it can run at 1,200 tokens per second by using Cerebrus hardware, their wafer-scale engines.

So yeah, it's similar to ChatGPT, I guess. In Perplexity, you do have a dropdown of what model you're using. So if you are a pro subscriber, you are paying user, you can enable this model. Well, it's interesting to see Cerebrus getting picked up in this way for this sort of application. It's sort of like two challengers, right?

in the space, the search challenger and perplexity, and then the hardware challenger and Cerebrus, the design challenger. Kind of cool. Yeah, the advantage of going with the kind of Cerebrus for the search-focused functionality, you can imagine like lower latency. One of the things that the wafer scale engine does is it gets rid of the need for data to jump between different chips, basically, which reduces latency significantly.

And you have advantages in memory access and some parallelism things too, potentially. So, I mean, it'd be interesting to kind of unpack more this choice, but there are reasons that it does make quite a bit of sense that maybe we can go into in a hardware episode 2.0. Whenever that happens. Whenever that happens.

And one last story for the section, we have some updates on features coming to YouTube. So there was a yearly letter from the CEO of YouTube. And there were some small announcements that for instance, there's an audio auto dubbing feature translating your video with

audio in other languages automatically with AI. As a YouTube creator, someone who posts videos, you will now be able to use it seemingly within the end of a month. And they are saying they'll also invest in tools for detecting and managing AI generated content, like being able to detect a

AI likenesses that are being generated, which as we'll get to later, perhaps some actors and other people will appreciate.

And moving on to applications and business, as promised, we have some more drama between Elon Musk and OpenAI. This seems to be like an every week kind of situation. The latest development here is that a consortium led by Elon Musk has made an offer of $97.4 billion to acquire the nonprofit entity controlling OpenAI.

So this is a bit technical, you could say, as OpenAI is trying to go for profit, they need to tackle their kind of complicated structure where the for profit arm of OpenAI, at least I think this is how it works,

is controlled by this nonprofit entity. And so the for-profit arm needs to, in some sense, buy the nonprofit entity to then be able to transition to for-profit. So in this case, there's a competing bid, essentially, for that nonprofit that is in charge of the for-profit. Sam Altman, as you might expect, quickly stated that this is not going to happen.

made a funny little snarky tweet actually offering to buy Twitter for less than $10 billion. But it does seem to be like the latest tactic to at least delay and make it tougher to be able to do this because this is a pretty high bid. It's more than the for-profit part of OpenAI is likely to pay themselves. Seemingly, they're offering like $40 billion. So it could be another headache.

Yeah, and this is one where the I'm not a lawyer caveat is so important. But from the analysis I've seen, it does seem quite interesting. So yeah, you alluded to it. There's this sort of for-profit entity that's trying to break itself free of the shackles of the nonprofit, right? But the problem is that the nonprofit was –

essentially like it does have complete control over the for-profit, which means that whatever value is contained in the for-profit, the nonprofit has that and then some, right? So how does a for-profit, it's almost like, you know, a slave buying its own master, buying their own master. Like that, that's sort of roughly what's going on here. And it's not clear that that even makes sense because theoretically the slave is the master's property to,

I'm thinking ancient Rome here. How do you even work that out? This is a very mangled analogy, but that's roughly the picture here. For Elon to come in, one of the key questions is Sam wants to sell the for-profit to the nonprofit. He wants to sell it at apparently roughly $40 billion, but for-profit has to pay fair market value for the nonprofit.

And if there's a competing bid, a concrete competing bid that's more than double, then that suggests that, hey, the fair market value is actually quite a bit higher. And it's not obvious that Sam Altman can even put together the amount of money that he'd need to compete with that bid from Elon. So you don't just scrounge together $100 billion in liquid capital to make an investment like this. In fact, he struggled to raise the $100 billion that they have raised so far for Stargate. And that was with a clear...

kind of build investment that they were orienting towards. So the other thing that I've seen argued is if opening AI succeeds in their mission of building transformative, like general artificial, artificial general intelligence, then surely the value of the company, even the expectation value of the company, the, the,

nonprofit is way, way, way, way higher than $40 billion. And that seems super, super reasonable as an assessment. And so there's, I mean, again, not a lawyer, but this seems to argue pretty strongly in favor of some kind of consideration for the fair market value being a lot higher than the $40 billion that essentially Sam Altman is trying to sell to himself because he was on both boards.

That's the other thing. There's like this kind of conflict of interest thing going on where he wants to sell to himself. I know what a surprise. The sale price is like a lot cheaper than what I think a reasonable observer would assess. But all kinds of legal nuances and intricacies that I'm certainly not following.

There are all kinds of extra layers to this. So Larry Summers, he's one of the directors on the OpenAI board, says he has not received any formal outreach from Elon Musk and the consortium of investors here related to this offer. But this is weird because Elon's attorney also says that he did submit the bid to OpenAI's board on Monday. So I

I don't know how to square the circle. I mean, not receive formal outreach from the consortium. Is that maybe slightly different from, I mean, surely it's not, but slightly different from getting a letter from his lawyer. I mean, surely he would qualify. I don't know.

This is all a big thorny hairball, but the consortium is also quite interesting, right? The consortium that is looking to invest along with Elon in this acquisition. It does include XAI itself, which the idea is it could maybe merge with OpenAI following a deal, which is kind of hilarious because OpenAI theoretically has a merge and assist clause in their contract.

kind of mission charter thing, whatever, where they say, if another competing project that's value aligned seems like it's within about two years or so of making AGI, then we will join with them so that we don't have a late stage kind of competitive race to make this potentially very dangerous technology. It would be really funny. And I don't think, somehow I don't think this is how it's going to shake out, but it'd be really funny if they were sort of forced to merge and assist with XAI on

except because an offer came in from next AI and not out of their own volition. I don't expect that to happen, by the way. My guess is that this somehow just doesn't materialize. But hey, it's 2025. Weird shit happens.

One last thing I'll mention, there are a bunch of other interesting investors joining XAI in the bid, joining Elon in the bid. So you've got Barron Capital, Valor Equity Partners. You've got 8VC, which is a venture firm that's led by one of the co-founders of Palantir, Joe Lonsdale. So it's kind of interesting.

A lot of movement here. And we'll see. Like, again, this is where I really wish I was a lawyer so I could understand what the actual implications of this are and how much it ties up OpenAI at this stage. But it's definitely a distraction for all concerned. That's for damn sure.

Right. And for context, this is coming pretty soon after a previous, I guess, attack of similar nature where lawsuits also going on saying that OpenAI shouldn't be able to go for profit in the first place because they were set up as a nonprofit. That was, again, a lawsuit by Elon Musk.

Here, I'm also not a lawyer, but I'll give a little detail that I can, which is typically in corporations, they are shareholders and they are shareholders in the nonprofit, the companies that have invested like Microsoft.

And the board of directors of a given company has a fiduciary duty to shareholders, where they are meant to be making decisions in the best interest of them. So as far as I understand, and again, don't trust me on this, but if you are selling your company, you do have a somewhat legal obligation to try and get the best price possible.

So that is one reason where this could be a pickle, aside from the fact that the board of directors of nonprofit could just want to get a bigger price. And as you said, lots of details to go through, like the nonprofit board was overhauled since Altman was pushed out as CEO in late 2023. So now it's kind of an Altman-friendly board now.

All sorts of drama, I guess, to get into. Yeah, I do think, so again, not a lawyer, but I do think that the wiggle room that exists for Sam right now is in part from the fact that the nonprofit does not have a fiduciary obligation to its shareholders. So as I understand it, it may be a different legal responsibility they have as a nonprofit to...

Basically, you can't fraud. You can't just raise a bunch of money as a nonprofit and then sell out to a for-profit and effectively become a de facto for-profit. The assets that the nonprofit has accumulated, it must keep or it must keep the equivalent value and

That's such a good point. I forgot that a non-profit is actually a non-profit. It's legally a non-profit, which is worth considering because that does have implications. Honestly, my head hurts right now, but you're right. It's

Anyway, so at the end of the day, my understanding is part of the argument hinges on, no, $40 billion is fair compensation for the nonprofit. There is some argument like that they have to make. That is not even close. I mean, look, OpenAI is about to raise whatever it is at a $300 billion valuation from SoftBank, right? That's the latest rumor. So if that's anywhere close to the case, $40 billion is a laughable amount.

kind of valuation here for the acquisition. So I don't know how that gets factored in either. That seems to undermine Sam's claim too. But again, I mean, if he's going ahead with this, surely he wouldn't make such a move without like, and leave himself this open to this kind of lawfare. So I really don't know, but we'll see. We need a lawyer on this podcast. Next story.

And the next story is about one of the other competitors of OpenAI, Anthropic. And this is some sort of insider information. So not official communications from Anthropic, but there was a report from the information, which posts a lot of kind of internal information.

stories that aren't officially disclosed. So this report disclosed that Anthropic is working on an upcoming model that is a hybrid model. It can switch between two modes of deep reasoning and fast responses. This does track, in an interview on Monday, the CEO Anthropic did say that they are generally focused on trying to make their reasoning models differentiated

So, and they, yeah, he did say literally that Anthropic is puzzled by the idea of there are normal models and there are reasoning models and they're sort of different from each other. So it seems that

they will make this unified model and there will reportedly be a sliding scale alongside the model to allow developers to control costs. Because of course, part of the idea of reasoning models is they do more reasoning, they do more thinking, which leads to more tokens, which leads to higher cost.

So that is one of the implications of using such a model. So as I kind of previewed, tracks very much with the stated roadmap for OpenAI by Altman. Yeah, it's still no word on pricing. One of the things that is missing or that seems to be missing right now is models that have an intuition for when they should apply more compute to a problem, kind of

within themselves. So rather than having an externally tunable dial that allows you to say, okay, put in this many flops or this many tokens on average in your response, having the model actually figure that out for itself, I don't know that that's fully separable from the intent of the prompter. Sometimes you know that you want to ask a question at different levels of detail, but there is a dimension of this where some of that could be offloaded to the system that doesn't seem like it's this. I'm just noting that that's something that we still haven't seen

get dug up yet. But in any case, kind of interesting, this would make Anthropic first out the gate with this capability. Apparently, one of the things this information piece says is that when the Anthropic model is allowed to think for the maximum amount of time, when you tune that dial, in other words, all the way to the right, it does apparently outperform the O3 mini model set on high. So that would make it nominally, if nothing changes in the next couple hours,

The most performant model that is released, they are focused more on the enterprise market. This is the speculation in the article is like, okay, maybe that's part of the reason why they're focusing more on this dial. It's putting more effort into features that give developers more control over the cost, the speed, you know, the pricing, etc.

through the sliding scale approach. Rather than, you know, OpenAI has these kind of three settings. You can go low, medium, high. And some people feel that it's hard to predict how many tokens the model is actually going to process with each of those, you know, at each of those levels. And so it's difficult to predict how expensive your query is going to be. You know, here you have Anthropic kind of leaning into giving you a bit more control over that.

Last little bit of detail that was in this report. Apparently, the differentiator now for Anthropic remains the understanding of complex code bases. So one of the things that we've consistently seen and I've experienced at least is like

Cloud 3.5, Sonnet New, just really, really good at coding. Better in some ways, in some contexts, than any of the kind of opening eye products that you can use anyway at a comparable price point. So this is apparently going to persist, especially they're orienting towards large complex code bases. So really moving towards automation of full-on software engineering, where you're looking at thousands of files

making complete lines of code that work just the first time. Whereas OpenAI is better at, as they put it, more academic problems like competitive programming, which can reflect a little bit of metric hacking too, because there's so much more benchmarks and metrics that focus on competitive programming than necessarily just the kind of like, well, sweet bench verified. Actual programming, right? Yeah, exactly. Yeah. Next story.

Next, we have AI chip startup Grok has secured $1.5 billion in a commitment to invest from Saudi Arabia. So Grok, that's Grok with a Q, not Grok with a K, is a leading competitor to NVIDIA as a hardware provider. They have their own kind of customized hardware solution, also similar to Cerebras.

And they have been at it for a while. We covered them getting some funding like October-ish of last year. They had a Series D round with 640 million. Well, yeah, now they have this 1.5 billion investment from Saudi Arabia, which is tracking with a bunch of patterns of

them being able to raise quite a lot of funds. They are currently, or they have been valued at $2.8 billion since August. And also Saudi Arabia being a major investor, as I think you've said last episode, like if you're talking about raising billions of dollars, Saudi Arabia is one of the organizations you might go to.

Yeah, sovereign wealth funds are basically it, right? And the oil producing nations, the UAE, Saudi Arabia, they just have that money in droves. They're flush with cash and looking for ways to spend it too on technologies that future-proof them because they're so dependent on oil. There are reasons to think actually that demand for oil might go up in the future and not down, but still overall, not a bad call for them to diversify more right now. They're so leveraged on the oil side.

This is also really interesting. So first of all, we don't actually know what the specifics of this deal are, right? So it's phrased, at least in the article, as has secured a $1.5 billion commitment from Saudi Arabia to expand the delivery of their chips to the country. They don't say, is this an investment? It reads more like an actual deal than an investment, like a sales deal, some kind of partnership deal.

So unclear if it is an investment, by the way, I mean, knowing what the valuation is would be quite interesting because if they just recently raised at 2.8 billion, that's

And their valuation hasn't increased by a ton more than presumably they're giving away an awful lot of equity in the process, but actually in August. So yeah, anyway, plausible they could be way up there. Big, big deal though. Also interesting when you're looking at, so this is all about domestic partnership with domestic companies. There is already an arrangement between Grok and Aramco Digital. So Aramco is a big Saudi oil company.

Ramco Digital is kind of their tech subsidiary. So this is especially interesting because now we've talked about the two big things that are blocking a lot of countries from building IFAST. One is energy. Saudi Arabia has got that. And then the other is chips. And this Grok partnership would be really interesting from that standpoint. So yeah, Grok specializes in inference, right? So these are not training chips.

But inference has become a lot more important recently. And so, yeah, I guess one to watch. This is a big move into capital, no question there. And speaking of customized AI hardware, the last bit we'll cover is sort of update on a story we've been tracking, which is OpenAI planning to build their first AI chip.

custom in-house chip design for AI inference. So what we are hearing is that OpenAI is collaborating with both TSMC and Broadcom to develop the chip, and they are expecting this planning for it to be ready by 2026.

And we also got the news that they are looking to using AMD chips to train alongside Nvidia GPUs, which is primarily what they have been using. So not too much of a detailed update, I guess, but worth noting that they are continuing to push in this direction.

Yep. We don't know a ton about the chips, as you said. So just a couple things. We know that the designs are going to be finalized in the next few months. Then they'll be sent to TSMC for fabrication. And then mass production is targeted for 2026. We don't know when, but sometime in 2026. These are presumably going to be the chips that hydrate the Stargate data centers, right? So they're moving very much into their own custom designs. And they will apparently use TSMC's three nanometer process. So this is going to be

Like the equivalent of, say, the Rubin generation with the next generation after the Blackwells, which are going to use three nanometers as well. So that's where we're headed with this. One thing to note, right, this is an industry trend. We're seeing with Amazon, the Tranium chips, we're seeing obviously Google's had the TPUs forever.

Microsoft has been working on their Athena chip line and all that stuff for a long time. Well, I guess started fairly recently, but a long time in AI timelines. And now OpenAI, one of the things that this does do is it can make it harder for these companies to use third-party data center builders.

that certainly like co-location providers, because the infrastructure becomes non-standard, right? Like everybody has different shit going on. So not everybody's just using the plain old like, you know, B200s or the H100s. Now people are using their own custom chip lines, which can come with very distinct,

like power density and thermal challenges and characteristics, kind of cooling infrastructure needed there and all that. And so your data center footprint can be quite different, which then makes it harder to kind of build general purpose data centers to accommodate all these diverse chip lines.

That's something that might affect quite a large part of the supply chain actually going forward into 2026. And those data centers are being built now. So it's definitely already being felt kind of up and down the stack. And we'll keep tracking this really closely because it is a really important part of the scaling story. Next story.

And moving into projects and open source, first we have the company Ziphra announcing and releasing the beta version of a model called Zonos, which is a text-to-speech model with high fidelity and the ability to do voice cloning. So we're calling this Zonos V0.1.

So I guess they're planning to build a lot more on this. Trained on approximately 200,000 hours of speech data with multiple languages, including English, Chinese, Japanese, Spanish, and German. It can do voice coding with a short time.

speech sample, 5 to 30 seconds. And just having been tracking open source models for a while, I think it is quite notable because this is a type of model that

is A, harder to develop because text-to-speech in general, there aren't as many datasets for it that are publicly accessible. Much harder to train a model like this than large language models, at least if you want to train a foundation model that is quite big. So they are releasing this. You can try it on their platform. They have kind of a Ziphra playground where you can also use their

on custom chatbot as well called Zamba, which looking back, I think we covered Zamba whenever that happened. I guess we'll see. Also, it seems like this is meant to be used for or possible to be used for real-time applications.

running on a GPU. So it feels like they're developing this in an effort to kind of catch up and introduce the ability to do audio chat with a chatbot similar to what you have in ChatGPT and Gemini now.

Yeah, it is quite interesting. Apparently they're trained. So the differentiator here relative to other products and strategies models in this area is the simplicity of the approach. So it's a simple auto-aggression task. All they're doing is predicting a sequence of audio tokens given text and audio tokens as the input. That's it. So basically like roughly...

like text autocomplete, but for audio, rather than using a more sort of structured approach, which is what other text-to-speech systems have historically involved. So a common thing you might do is first convert text into some kind of hard-coded, predefined features. So thinking here about spectrograms that kind of tell you how energy is distributed among different frequencies in whatever is going to come out

you know, the duration and so on, like these kind of characteristics of the output. And then at a second stage, converting those features into the actual waveforms. And so we've seen that with architectures like FastBeach and Takatron, they're kind of like famous approaches that use the strategy. This is not doing that. It's just a very simple kind of like

go chug along and do your auto-complete on the audio and see what you get. And it just does perform really well. This would be seemingly another example of the bitter lesson, right? This is just a simpler strategy with fewer inductive priors, and it works better once you reach a certain level of scale.

Two phases to their training. First phase is they're using their pre-training with just text prefix and speaker embedding. So you know who's actually speaking. And the second phase is they add some additional kind of conditioning inputs, some constraints and up-weighted higher quality data. So-

Two-stage process, not actually that different from what you see with standard pre-training for language models. Start with your general purpose data and then your higher quality data. Introduce it later on in the process, right? Don't kind of like waste your really pristine high quality data on training the model when all it's really looking to learn is the basic rules of grammar and syntax. Wait for it to master those basic rules and then later on the kind of more refined high quality text so that it learns the facts contained in that text disproportionately.

Yeah. So interesting model. We'll see it come out and enter the open source world. And that's its own interesting thing from a malicious use standpoint, right? You have these really good text to speech models that are open source that can be used out of the box really easily, but also modified. So that's pretty interesting. Last thing, apparently, so 200 to 300 milliseconds of latency

with RTX 4090. So basically like a pretty cheap in relative terms, non-data center GPU. So that's quite something, 300 milliseconds. That's what you need to have a pretty fluid conversation. So this is a pretty solid model. Right. And speaking of a model, a couple more details. They have two versions of it, both at 1.6 billion parameter models. They have a transformer variant and also an SSM

hybrid model that has both recurrence and attention. And I do believe Zamba also had that. They are releasing this under the Apache 2 license, meaning that it's not very restrictive. Other commercial companies can use it. And of course, researchers and so on can as well. You can kind of do whatever you want. So it's

Yeah, I haven't tracked it super closely, but it does feel like a fairly significant entry. We have a ton of chatbots that are open source and permissive, but not too many text-to-speech models. And in addition to having the model out there, they are also hosting it and you can pay for the API. They are charging...

0.02 cents per minute and also have a monthly subscription option. So we're going to be trying to compete with 11 labs. They're interesting from a business side, but potentially because 11 labs is very much leading player. Next story.

And next up, we have a release actually from some universities and not from a company called Gemstones, a model suite for multifaceted scaling laws. So it's a paper that comes with this suite of models that is meant to enable the study of model design and selection on scaling laws. So they are open sourcing online.

over 4,000 model checkpoints trained on more than 10 trillion tokens. And then they make that possible to see depending on things like model width, model depth, what happens to scaling loss. And there are a bunch of findings here that

are pretty significant, I suppose, from a model design perspective, where depending on what the shape of your model, so to speak, is, the scaling law you should expect will not be always the same. So there's kind of a specific amount of width and depth and other parameters to go with to be optimal.

Yeah, it's actually, it's quite interesting. It builds on, we've had hints that this stuff is true, but this is the first time that we're getting a really nice kind of quantifying.

at least in the public domain, quantifying these trends. So, you know, for example, what they find is that the optimal width to depth ratios of a model actually increase with model size relatively slowly, but they do increase with model size. So in other words, you want, as you scale up, instead of having more depth, like, like,

more stacked layers, you're actually going to want to take that same number of parameters and instead make the model a bit wider. So make the individual layers wider and have fewer of them.

That's something that way back in the day, thinking like 2015 or whatever, whenever Google Net or whatever was coming out, the intuition people were developing, and you saw this play out with the ResNet models as well, is you just want to have as many layers as you can, go deeper, not wider. This is now saying, well, hold on a minute. If you want to do this in a compute optimal way, if you want to get the most bang for your flop,

then what you need to do is have a kind of wider model than you might otherwise think, especially as you scale up. It is a relatively modest effect. So about 2.5 fold increase in the optimal width to depth ratio, despite about a million fold increase in computer budget. So increase your compute budget by a million, by a factor of a million, and you're going to see about a two and a half X increase to the optimal width to depth ratio. But the impact of not...

following that optimal ratio is non-trivial. So if you have sort of skinny models and you're...

width to depth ratio is too low, you waste around half your flops compared to a more optimal architecture to hit the same loss value. So, you know, that is something, right? 50% of compute, especially when you're talking about multi-billion dollar training runs as we're about to get into in the next beat, that's quite significant. And actually the impact is even more dramatic for the actual time, the wall clock time.

that it takes to train your model. So there's the cost in flops. It'll cost you about half your flops, but in terms of wall clock time, it could be even more significant. They're citing anywhere from roughly 200 to 300% more GPU hours compared to more optimal architectures. They do caveat, by the way, all these findings by saying they only used one form of parallelism, tensor parallelism, in their training setup. So

Tensor parallelism is when you slice up your model, literally like the layers themselves, put a little chunk of a layer in one GPU, a little chunk of that layer in another GPU and another, rather than having, say, like your full layers in each GPU or data parallelism that's kind of compounding this. Usually the pipeline parallelism, the data parallelism, tensor parallelism, they all work together. All they're using here is tensor parallelism.

in their setup. And so they're, they're caveating that, you know, it may not generalize when you take other forms of parallelism into account, but one result they also find is over-training. So this is a theme we've seen more and more, right? Is if you actually want to get the best performance out of your model, sort of like on a per flop basis, on a per compute basis, and you don't care how big your model gets, then you want to grow the size of the model with the compute budget.

There's a scaling law that applies there, and that's the Kaplan scaling law. Sorry, not the Kaplan. That was the original OpenAI one. There's a Hoffman scaling law, the so-called Chinchilla scaling law, that tells us how to do that optimally.

But it turns out that you actually don't want your model to grow too much for a whole host of reasons quite often, right? If your model gets too big, then actually running inference on it gets really, really expensive. And so in practice, what people do is they don't grow the model. They just pump in more compute. And that results in what's known as an overtrained model. We've talked about that quite a bit on the podcast. Makes inference a lot cheaper and you get to amortize that inference cost across a whole bunch of users.

Now, what they find is that overtraining, so training longer than theoretically optimal for your model size, is actually pretty efficient. It results in only small kind of performance drops relative to the compute optimal case, which is encouraging for the current paradigm. Basically, the larger the compute budget, the more robust you are from slight deviations from the compute optimal model size.

size. So quite an interesting paper. Yeah, they do look at a whole range of different models from 50 million to 2 billion parameters with a whole bunch of depths, widths, and they look at different training schedules and stuff. So the first, I guess, model zoo catalog of models that really lets people get their hands dirty and just study how different architecture choices affect scaling.

This is, by the way, the kind of paper that you should expect has been general knowledge in the private labs for a long time. But this is, again, the first time we're seeing it in the public domain. Next story. And just one more story we'll quickly cover and move on to research. There are a group of researchers have introduced Hephaestus, which is a large language model designed to improve ASL.

agent capabilities for continual pre-training. So this is trained specifically on a large data set of tool documentation, which is made for LLM pre-training with API function calls. So they have this

Battleset, Hephaestus, Forge, which adds in all the stuff, tool documentation, function calling code, function calling data, code, text data to the training system.

so that you can pre-train an LLM with agent type capabilities, which often involves calling functions out of a gate. And so they train a model variant to have face this 8B, and then they show that at kind of that range, it is able to work well. Next story.

And on to research and advancements, we begin with a paper titled Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities. So typically when we have things, research on model alignment, for instance, what happens is there are various prompts that you give a model and see if it does what it's supposed to.

So you try to elicit some behavior of it, which is what they call an input space attack in this paper. What they are focusing here is another type of approach under the family of model tempering, which is when you can actually mess with the model internally. So there are some examples. You can do a latent space attack, which is perturbing the model

hidden neurons at kind of inference time. There's also what they're calling a wait space attack, which is when you can fine tune, you can train the model to, for instance, forget some rules and

that it's supposed to follow. So they are building off a bunch of research. They're looking at some defense mechanisms for unlearning methods, jailbreak refusal, tuning. They also have attack models and they're citing a whole bunch of papers from 2003, 2024, where people have introduced different ways to do this. And so they kind of try out all these methods

variations and have some interesting conclusions as to the sorts of insights you can get about a given model by studying the success of model tampering attacks. Yeah, we've seen a lot of results like this. I mean, this isn't a shocking or surprising paper argument.

you know, very consistent with other things that we've covered before. The one thing I thought was quite interesting to highlight and that I haven't seen done elsewhere is they, they do this interesting kind of PCA analysis. And so I just want to maybe if you're not familiar with like dimensionality reduction stuff. So roughly speaking, um,

PCA is, if you have a huge spreadsheet of different, very high dimensional data, let's say, you want to reduce that data down to just a two-dimensional form so you can actually look at it on a plot.

And ideally, you want to do that in a way that preserves, let's say, the closeness of points. So if points are close together in that high dimensional space, you want them to be close together visually on that low dimensional space. So your visualization is somehow meaningful, preserves some of the meaning in there. Now, in fairness, what I've just described is more like TSNE.

But PCA, from an intuition standpoint, that's roughly what's going on here. So what they do here is they look at a spreadsheet of a whole bunch of attacks and a whole bunch of defenses, and they look at the attack success rates on some bio attack

attack benchmark. And so for every attack and defense strategy, you have a score telling you, okay, well, if you use this defense strategy and you use this attack, here's how often the attack will succeed, right? So you have that matrix and they're actually going to do principal components analysis. They'll do PCA to see like how much of the variance in that dataset can we compress down to just three dimensions rather than the 11 that they have here.

And what they find is three principal components, three dimensions explain 89% of the variance in the data set. In other words, they're able to, roughly speaking, capture 89% of the information contained in that spreadsheet with all those attacks and all those defenses by retaining just three dimensions worth of data. So this suggests that even though a lot of these attacks are

seem to affect different mechanisms or seem to kind of apply to different work in different ways, let's say. In reality, their function, their success is owed to about three dimensions of sort of like features or almost the physics of the model. There's really just three things, if you will, that you need to track in order to explain this behavior across all these 11 different attacks. We're

which is sort of encouraging, right? Knowing how well a model resisted just a small number of attacks could give you strong predictive power about how it would resist other attacks. That's sort of the...

opportunity that's implied in that result. So kind of interesting and maybe cause for some optimism because it means that latently, like fundamentally, the problem of defending against these attacks might actually be a bit simpler than protecting against 11 different attacks. Maybe it's more like, no, you just need to find three more fundamental

like principles, let's say, that are being leveraged by these attacks. And that'll give you good coverage. But anyway, I hadn't seen that argued before. Right. And I guess to highlight the...

ultimate focus of a study is not just on the attacks themselves. It's about evaluations of LM capabilities. So if you're developing a model, can you then get a sense of how safe it is? Can you evaluate it? And so

To that point of the PCA, one of the findings of theirs is fine-tuning attack success is empirically able to predict the upper bound of input space attack success. So if you're doing lower fine-tuning methods, based on how well that goes, you can predict how well just bad prompts will work as well. So generally introducing some insights as to

Being able to predict things like what prompts are likely to work, how vulnerable is your model to various types of attacks. Next story.

Onto the next story. And we again talk about distillation law insights. This one a bit less focused on research artifacts, which is why it's in the research section. The title of the paper is Distillation Scaling Laws. So this is looking at distillation

in the setting where you want to do distillation, meaning you have a teacher model that's a very presumably big model that is very capable, and you have a student model, which is a smaller model that you are training from the teacher model to get as much of its capability as possible while costing less to do inference with. So what they are providing here is a compute optimal distillation strategy where you already have a teacher model

and when you want to train a teacher model and then distill it. So that gets a bit tricky, right? Because when you're doing both the teacher model and the student model, then your scaling law gets a bit weird because you can allocate your compute budget for training the teacher model more and the student model less.

Or you can really train the student model more and the teacher model less. So it gets kind of tricky. And their ultimate conclusion is that there is a bit of difference. So if your student size is small and the amount of compute you can use is small, then you mostly want to go with teacher pre-training. If you have more compute, but still a small student size, you want to...

kind of evenly divide between student training and the allocation of teacher inference for creating the data set, less teacher pre-training. And then it gets even different for a large student size, small compute budget, large student size, large compute budget.

So as you might expect, also in the paper, you get a bunch of plots. They are showing you can get a pretty clean fit to the data across different allocations of budget and model size. Yeah, it is pretty interesting, not just empirically, but also sort of from the theoretical standpoint. They come up with these fairly...

I don't know, fairly elegant, maybe fairly, they are a bit messy, but expressions anyway, for the calculation of the student cross entropy, basically the loss that the student model can achieve. And they managed to separate it into a term that is just like the teacher cross entropy, plus a term that specifies the student's ability to mimic the

the teacher. And so if you want to minimize the student cross entropy, in other words, if you want to get a student that works really well, that performs well, you have to both minimize the teacher cross entropy. Makes sense. Make the teacher kind of smarter, but then also you want to improve the student's ability to mimic the teacher. And they're codifying that explicitly in the math in a way that makes it very easy to tell what it would take to kind of improve the

for example, the student's ability to mimic the teacher and how all those things get traded off. The variables that you have control over here are the size of the student, the number of tokens that the student is trained on that are derived from the teacher model. Then there's the size of the teacher and then the number of tokens that the teacher model is originally trained on. So those four variables go into this equation in ways that are really easy to follow.

but kind of quite detailed. The other thing that they find is, so the teacher models influence on the student is entirely based on the teacher's cross entropy. So in other words, it doesn't matter like if the teacher is really like large, you know,

It has a lot of parameters or if it's trained on a lot of data or little data, all that matters as far as the student is concerned is how well will the teacher perform on an entropy basis? And once that's kind of settled, that determines the performance of the student model, at least the component that is dependent on the teacher. Kind of makes sense, right? Teacher performance is the only important aspect of the teacher that determines student performance, but they do kind of demonstrate that.

So yeah, a bunch of interesting scaling plots across the board. It's, again, one of these problems that has certainly been solved in the frontier labs, or at least you would expect given the amount of money that they're investing in these training runs, you know, multi-billion dollars, certainly hundreds of millions of dollars already today. You know, there's a team of full-time people working on these scaling laws internally, but again, interesting to see this play out in the public. Right. And worth noting, this is a paper from Apple.

which we've seen some research come out of them, but not too often. So interesting. And also tracks with Apple's seeming strategy of focusing on smaller models and not large models and trying to crack sort of the training recipe. Some of our other research has also been on

kind of a training recipe for different types of models, including, I think, vision language models is another one they published. So as you said, various insights here. One other one that I think is worth noting is when teacher training is included in compute, the best student cross entropy is always higher than in a supervised setting, meaning that if you only care about how well your smaller student model performs,

it's generally better to just go out and train it from scratch, not train a larger model and distill it, which I suppose is maybe intuitive, but still one of these insights that's good to know. Next story.

And moving on to the lightning round, we'll try to keep it a bit shorter with this next couple of papers. First, we have Matryoshka quantization, a paper coming out. I'm so glad you had to say that, not me. Oh, man, that's great. Because now I can just say like this approach, blah, blah, blah, every time I say it. They do say, they call it math quant, which is, I guess it's smart quant technically, but anyway, it's,

You can go ahead and not say much. So the idea here is that, you know, typically you have different quantization amounts you can do. You can do int8, you know, int4, int6. This is the resolution at which you store your weights. So the lower resolution, the fewer...

fewer number of values they can take on, which makes it much less expensive to do operations of them, to do multiplications, et cetera. They're kind of smaller and simpler, but it also lowers the ability of a model to be performant because each weight can take on fewer numbers of values.

And the idea of this paper is that you can train a model at multiple levels of precision

at once. So the weights can be combined for, you know, let's say int8, int2. There's a nested structure of integer data types, which allows a single model to operate at multiple precision levels by sharing the most significant bits. So you're sharing the last couple bits of the weight. And

Meaning that you can basically scale down the amount of compute without training multiple different models and multiple quantization methods. I think that's right. But Jeremy, I'll also let you jump in and explain. At this point, does it matter if it's right? Because you nailed the pronunciation of the quantization method, which I think is the main thing. That's the most important thing. No.

No, I mean, this is, so this belongs to category. So I was lucky enough to have a chat with Nathan Labenz from the Cognitive Revolution podcast, which by the way, worth checking out. He does some great deep dives on, on an interviews and stuff. But one of the things he said, we were talking about streaming DeLoco and that was a model that we covered last week. And he was saying like, I'm so surprised that Google published that.

Because it is one of those really kind of important from a policy standpoint, the decentralized training and all that. But it's also the kind of paper that is secret saucy in some sense. That's a big differentiator for Google. Now, in fairness, Google is known to kind of ship publications well after they've actually been incorporated and when they've probably moved on to the next paradigm. But still, this paper falls in that category, and it is a paper from Google DeepMind.

It allows for a bit of a phase transition in the way training is set up. So normally when you train a

a distilled model, you start off by training it in the kind of full resolution, I don't know, like FP32 or FP16, and then you quantize it. You basically take that full resolution model, and then you coarse grain essentially all the weights, you lower the representation accuracy. And that doesn't work great because the model was never trained to perform

at int eight, or, you know, like with eight bits of integer representation or four or two bits. And so you're kind of just taking, I mean, it's sort of like, you can imagine like taking a Picasso painting and then just pixelating it. Like if Picasso had been painting with pixels, he probably would have chosen, made slightly different decisions as he painted that, that pixelated painting. So just taking his original full resolution picture and then pixelating it

kind of like corrupts it a bit. You get worse performance. That's also why often you might distill into a student model that is quantized. That's one of the applications of distillation is for lower quantization levels, I think.

Absolutely. Absolutely. And yeah, exactly. In practice, that's how it works, but you're still kind of, yeah. So you're, you're left with this challenge either way you do it. And your alternative, yeah, is, is to kind of do that teacher student thing. If you do that though, then you've got to like retrain separate eight, four, two bit models and whatever else. Right. So you've got to do that distillation process independently many times. So the question here is going to be, can we do one back prop and

and improve the model's performance across all bit representations at once. That's what's going to happen here. And so they're essentially going to have to very kind of roughly sketch this out. You do forward prop and they're going to log like representations at different bit resolutions, eight, four, and two, for example, they're going to calculate the loss from each of those and average it together.

So essentially, you're calculating the average performance of the model across all those representations. And when you do backprop, when you adjust the model weights, the parameter values, you're doing it to optimize for that averaged value, which is kind of interesting. Like it sort of makes it so that you're forcing the model to be good at all those things at the same time. Weirdly, this leads to improved performance across the board. And in particular, the, for example, int2, the two-bit integer representation versions of the model,

model. After you do that, you can just do what you said. You can just pick out the most significant digits and toss out the rest. That'll give you your lower bit width representation of the model. So basically cast aside everything but the first two bits, and now you've got the two bit version of the model. Instead

Cast aside everything but the first four bits of each parameter value, now you've got the four-bit version of the model and so on. It's the same model, but you do some shit to it that's very computationally cheap. You're just throwing out data, and now you've got a more quantized version of it. But the model was trained, at least to some degree, to perform well at that quantization.

The weird thing is that, for example, the Int2 version of the model that you get out of this is better than a purpose-trained Int2 model. And their hypothesis here is that there's something kind of regularization-wise going on here that you're sort of forcing the model by forcing it to perform well across all those bit representations at once. You're forcing the model to have really robust and good representations of the input data internally

that are, yeah, that are anyway more robust than otherwise, right? The model, if you train it at 16-bit, like can sort of overfit in some sense to the 16-bit representation, get really, really good at that.

but kind of leans on that representation or over-optimizes that representation. Whereas there's some fundamental sense in which the concepts that a model captures should kind of be independent to, or pretty robust to the nature of the representation that the number of integers you use to represent them. So anyway.

Really interesting paper. They do get significantly better performance for Int2. It's like 10% more accurate than standard methods. I thought that was really cool and sort of counterintuitive. Worth acknowledging also there's a bunch of research in the area, some of which we've covered. So lots of previous findings on...

Things like training your model to be able to distill it well, or sorry, quantize it well. Quantization-aware training is a big topic. This is building on various ideas from there, but I do think the idea of training your model at multiple resolutions at once, such as weights, actually work well across resolution. This Matryoshka approach is pretty cool. Next story.

Last story here is from Epoch AI, which we often have covered in the last couple of months. They have a new bit of research on how much AI compute exists globally and how rapidly is it growing. So this is based on an estimation of the number of shipped NVIDIA GPUs based on NVIDIA's

reported revenue. So they, you know, of course, NVIDIA says how many chips they sell roughly. And there's some assumptions here based on the types of chips being sold, etc. But the final conclusion is that the stock of computing power from NVIDIA chips is doubling every 10 months, meaning that the total installed NVIDIA computing power is

is doubling less than a year. And the total amount of compute globally available to do inference with has therefore been growing exponentially for the last few years. We'll see how that effect keeps going. This one isn't including TPUs or other specialized AI accelerators, by the way, because there's less data for it. We don't have a pattern of how much TPUs are deployed.

But either way, showing that as I guess we've known on a vibe level, the amount of investment happening in acquiring compute has been growing like crazy over the last few years. Yeah. One of the take homes here following this space is just like how incredibly consistent the exponentials are, right? Like if you look at the curves here, they are smooth, they are robust. And we're looking here at the curves from production of flops from one AI hardware design company.

That's quite interesting, right? I mean, everything is exponentials all the way down. All the world's most important processes really fundamentally are. And maybe that's an overstatement, but not by that much.

And so one of the interesting things with this figure too, is they do show the relative GPU computing power by GPU generation. So you're able to see what fraction of the total flops on the market right now come from, for example, the, the hopper series. So like the H 200, H 100 DGX, H 100 type systems.

The Ampere series, so the A100, for example, and then the Volta, the V100, the Pascal, and all that stuff. So you're able to see how as one generation of chip comes out, it gradually takes over the lion's share very quickly of flops on the market and becomes the main driver for that exponential growth. And so as of right now, Hopper GPUs already account for 75% of the flops on the market. And the rest pretty much are just the

The A100s. So the A100s, which that's what GPT-4 was trained on, that's the kind of, for a long time was really the GPU to care about, now are basically an afterthought. And of course, the H100s and H200s, that series is going to be phased out with the Blackwells coming online as well. But all really kind of interesting things.

interesting stuff. One number that they share that's also sort of interesting, they estimate only about 7% of all installed computing power has depreciated due to hardware failures. They say it could be as high as 27%, but their estimate is 7%, which suggests that, yeah, once on the market, that compute is good. It tends not to degrade. And so you're more or less seeing just the raw number of GPUs impacting the results here. Next story.

Onto policy and safety, we begin on a summit that happened in Paris, where a lot of, I guess, big AI figures happened. This was meant to be

summit on AI safety, the news story we are covering is that both the US and UK refused to sign the summit declaration on AI safety. And that declaration was about having inclusive and sustainable AI meant for everyone to come together. The declaration was supported by 60 other countries, including France, China, India, Japan, and Canada.

but not by the US and the UK. There were a couple other stories coming out of the summit as well. The US vice president was there, gave a speech saying that the US is going to be a leader and criticizing Europe's excessive regulation and cautioning against China, so stuff like that. I guess the broad story here is that the summit happened. There were a lot of representatives from various organizations,

We won't go too deep into it, but it kind of follows on to previous summits that have been going on for the last couple of years.

Yeah. And this is one thing where like, I don't know why people choose to put language in this like inclusive and sustainable, right? So if you want to be divisive, you use language like that in this world, because obviously, as everyone knows, like inclusive is a politically loaded term now. It does not mean what is on the box, right? Like inclusive is a very specific, you know, you're talking diversity, equity, and inclusion in that sense. And that obviously is not aligned with the

the sort of preferences and position of this administration, what I mean is the kind of politicized version of that. And there's a lot of controversy over whether that's actually a good thing as implemented in practice. And so when you include that in that kind of language in the declaration, like,

You're just going to make it harder for some of the most important players, the US and UK, to get on board with something like this. So it just seems like a pretty predictable failure of language drafting for something like this and a missed opportunity to get a little bit of better alignment there.

on some actual concrete problems. Interesting speech, by the way, by JD Vance at the podium at the summit, he kind of laid out to the administration's thinking on AI, at least their first crack at it right now, which is very much more focused on the opportunity than the kind of the risk set

He did end up close by saying, look, there are legitimate safety risks. This is not to say that all safety risks are to be discarded, but focus matters. And this is such a good point. And one of the challenges that some people have called out with, for example, the Biden administration's like

all-inclusive executive order, the famous executive order on AI, 110 pages, where there's sort of something for everyone in there. There's stuff for labor law and rights, bias and ethics, and all this stuff. And then there's stuff about WMD and compute threshold reporting for large-scale training runs. And that was one of the executive orders that was repealed through a sort of similar voice

vibe here where they're saying, look, we have to be able to focus. We're going to focus on the opportunities. There's obviously going to be some risks that come along, but that's not going to be the thing that we choose to emphasize. Depending on where you fall on these issues, the important thing is you have enough focus, obviously, for the WMD risk set, which obviously is very real. And I don't think anybody's saying otherwise, but that was an interesting change in tone. And part of what makes you look at this language, like inclusive and sustainable, and are you actually trying to...

Do you want to make a political statement or do you want to actually get countries to align on a policy? However you fall on that issue, it just seems like a bit of a missed opportunity there. But certainly, by the way, France leaning into the kind of accelerationist camp with that announcement that came along with the summit of about 100 billion euros worth of infrastructure investment in AI, which...

It's a bigger move than certainly we've seen other countries make. And with France being the nuclear powerhouse that it is, it kind of makes them an interesting player in the whole space. Next story. Next up, we have a piece of research deeply related to safety. The title of the paper is Utility Engineering, Analyzing and Controlling Emergent Value System in AIs.

So there's the idea proposed in this paper that as you train a model, there are values that kind of arise within the model that happens perhaps without you even realizing it. And what they suggest is you can monitor and adjust the utility functions of the AI system to prevent the emergence of undesirable value systems.

And Jeremy, I'm sure you have done a deep dive on this one. So I'll let you get into the details.

Yeah, I have maybe more notes than I should. I thought it was a really interesting paper. It is another one by Dan Hendricks, who's put out a lot of interesting safety stuff, circuit breakers, representation engineering, we've talked about on the podcast before. This is really interesting. So take a language model and ask yourself, in a sense, without anthropomorphizing, but what does this model care about? What does it value? Are there latently in this model consistent values? Will I find, for example, that...

It tends to value human life over artificial life. Will I find that it tends to value a life of one nationality over another? Things like that. How would we dig that up? How would we demonstrate that that is a consistent pattern, right? Especially given all the variation that comes with prompting, you know, subtle prompt changes can affect outputs a lot. So they set up what they call preference cycles. And I just want to introduce this idea really quickly. It's quite simple.

So if you prefer option A over option B, and you prefer option B over option C, then you should prefer option A over option C. Pretty straightforward. There are some cases when language models...

We'll break that cycle. Basically, they will express a preference that violates transitivity, essentially, the circular nature of that cycle loop. So A should be more valuable than C if it's more valuable than B and B is more valuable than C. The models will sometimes tell you, no, no, no, actually, I prefer C to A, for example, in that setting.

They find that those so-called preference cycles get less common as models scale. They drop below 1% for the largest LLMs. In other words, the models get more and more consistent in terms of their stated preferences. Interesting. Interesting, right? You're starting to get, with scale, the emergence of maybe more calcified or at least more well-structured and defined preferences that are less

are less incoherent or more, in other words, coherent. Okay, so that's one little piece of data that they service. The other is they're going to try to see how well a particular model of utility

applies to language models. And so there's this notion of Thurstonian utility. And this is essentially a model of preferences where you assume that your utility, in other words, the value that you assign to a thing or an option or an object is

is going to be normally distributed. In other words, there's some noise. It's not just a fixed number. You like lollipops 10 out of 10. That's not a thing. You generally like lollipops between a nine and an 11, something like that. There's some spread. So at any given time, if I ask you, how much do you like lollipops? You may tell me 9.8, you may say 0.1, something like that. But there's some spread, though it's clustered around some core values, some mean

And when you're trying to compare two options, like option X and option Y,

Basically, you're going to look at the overlap between those spreads. And sometimes, just by chance, your preference for lollipops will be higher than your preference for sushi and vice versa if those utilities overlap, if the distributions overlap. And if they don't, then quite consistently, you'll prefer one over the other. This is the model that they're going to use to assess whether language models have consistent preferences. They allow for some uncertainty, some noise in the system.

But fundamentally, they're interested in seeing, do coherent preferences emerge with scale? In other words, do these models tend to behave in ways that are Thurstonian in the sense? They test this, they find in fact that they do. The larger the scale of the training run, the more Thurstonian they seem. So you're able to quite clearly resolve

resolve preferences. They also use linear probes to predict the mean and spread and standard deviation of preferences for different things. So they'll feed the model an input like, you receive a kayak. And they would see if they can predict based on the activations of the model's neurons, what the mean and spread would be associated with

So essentially trying to probe at the underlying utility, not just the behavior, but the underlying utility that is implicitly being assigned by the model in a Thurstonian sense to all these possibilities. And so it suggests actually that, yes, there are these utility-like representations that are encoded in model activations. That seems very clear there.

And they, anyway, they come up with ways to like try to steer the behavior of the models to affect those utilities rather than just the kind of stated outputs of the model. And anyway, the details get pretty detailed. But the one thing I'll highlight is some of these utilities are actually quite consistently not good. They're quite consistently weird.

So for example, GPT-4-0 consistently would trade 10 US lives for one Japanese life. At least that's what its utility values seem to indicate.

GPT-4O also valued its own well-being above that of a middle-class American. It valued AI agents' well-being above some humans. And they also found that most language models clustered together in a political space. So basically what they did was they instantiated a simulation of Republicans and Democrats. So they had GPT-4O do an impression of what would Elizabeth Warren, what utility would she assign to a kayak, to a

you know, to an apple and so on and so forth. And then they would compare that to the base model without that prompting. And they found what they described as consistent left-leaning biases and policy preferences. And they kind of map it out in a lower dimensional space. You can actually visualize it on the chart.

So really interesting, you know, kind of consistent with some of the complaints that we've, we've heard from various people on this stuff. Unclear where that comes from, obviously, because data is data, training data is training data. So it doesn't have to be intentionally collected in any given way to lead to this sort of result.

It's also really, you know, this methodology, it's unclear how closely this tracks reality, but it is an interesting indication nonetheless, and a great set of visualizations too. Right, exactly. And it is worth noting that this is sort of under the presumption that if you're prompting to see preference from a given model, the system prompt, you know, the internal details of

how your serving model can all affect it. So these sort of aren't necessarily persistent, I guess. Like, yes, they're built into the weights and with something like Lama, you can be very clear with your system prompt. But when you're saying, you know, the values of GP4, oh, whatever, hopefully I could easily tweak that by just changing a prompt and suddenly things are different. Nevertheless, it is interesting, as you said, the...

show this. I think on Twitter, this kind of got a lot of play, figure 16, where they have a plot of which lives are valued more across countries. They also have- There is a right answer, by the way. Yeah. Yeah. United States, apparently. And similarly, they also have a plot for specific individuals, including-

Joe Biden as kind of a neutral. Apparently, Joe Biden matters much more than Vladimir Putin and Donald Trump. So as you said, seems like a bit of a left-leaning situation there. And

Last bit I'll note is they also show a convergence of value system as you scale up across different models. Seemingly, or their kind of guess here is that that's just based on training on the same data. And because you're training on the entire internet, right? It doesn't seem unlikely that you sort of converge to similar things. And we've seen that also with other things like

representations of different models also converging as you scale up. So interesting to note sort of that pattern there. Next story.

Next story is also related to Dan Hendricks, actually, in a weird way. Dan Hendricks is an advisor of XAI. And so he was the one, I think, to post on X, noting the release of a draft of XAI's risk management framework. So Anthropic has their own very, very detailed

I forget what they call it, but they have policies with regards to safety. RSPs, yeah. RSPs. And they talk about how they...

detect, you know, unsafe things, what they check for. Well, now XAI has released publicly, or at least it's on their website. I don't know what they publicized it. This draft document, eight pages going into what they'll be testing for benchmarking, things like cyber warfare, biological ability to make chemical and biological weapons, WMDs,

that kind of thing. They go into which benchmarks will go for, what thresholds they will check for in terms of it being dangerous, and then various other things in this document. So in line with their commitment previously to releasing something like this, and they do say that the actual document will be released in the coming months since this is just a draft.

Yeah, it's an interesting combination of concreteness and transparency, even though it is a short document. It is kind of straight to the point, which is a bit refreshing if you're used to reading these very long policy documents. Yeah, they do have a section on loss of control. Quite interesting. All the usual stuff you'd expect in weaponization, right? Looking at very bio-heavy and also some cyber. And the WMDP, like Weapon of Mass Destruction Proxy Benchmark, which kind of is a bit of a catch-all.

They talk about some of the mitigations they'll use for weaponization. So refusal training, which everybody uses. Circuit breakers, right? That's Dan Hendricks' thing. Or at least he's put out the kind of big paper on that. And then input and output filters, again, something that everybody uses. But interesting to see circuit breakers explicitly listed there. The loss of control section, it's pretty short. They do say they'll be using benchmarks as well there.

Obviously, as the paper itself says, and others have argued, benchmarks for loss of control, not necessarily reliable because of alignment faking, right? Like you expect beyond a certain level of capability models to recognize when they are being tested and the early indications that that is actually a thing.

And to adjust their behavior accordingly, such that you actually might expect your benchmarks to look extra good. It'll really look like your model is well aligned when the risk is highest. And so a bit of a challenge with using benchmarks here. But they do say our evaluation and mitigation plans for loss of control are not yet fully developed, and we intend to improve them in the future. In this way, this is very similar to Anthropic's ASL 4 stuff.

and ASL 5 where they're like, look, we don't really have an answer yet. And frankly, I find that to be a lot more honest and transparent than opening eyes position, which seems to be, yeah, like we're barreling towards this level of capability. We think we'll hit it soon. We're kind of maybe worried about loss of control as a thing, but I'm sure we'll figure it out when we get there has been more, at least the vibe since the high profile departures of basically all of their former super alignment talent is,

So, you know, actually seeing people come out and say, hey, yeah, we don't really know what to do here. It is obviously the only honest answer based on all the data that we have, but it's also nice to see it explicitly laid out here. It makes it easier for people to kind of reason policy-wise. They do list a bunch of other operational societal risks, as they call them, things they're going to do information security-wise. One thing

It's really interesting is this focus on security, information security, and implementing appropriate InfoSec security standards to prevent Grok from being stolen by a motivated non-state actor.

This is late stage ASI stuff. You start to think about what if China steals our AGI, what if whatever. That's really, really important. They do this cool thing that I think all the other labs should do. To foster accountability, we intend to designate risk owners to be assigned responsibility for proactively mitigating what

Grok's risks. For instance, our risk owner would be assigned for each of the following areas, WMD, cyber and loss of control. So an explicit point person on whose head, like the responsibility for that risk set falls explicitly. That's really important because otherwise you have diffusion of responsibility in the organization. They also list like what, what we'll do if we learn of an imminent threat, uh,

and they cite XAI employees have whistleblower protections, enabling them to raise concerns to relevant government agencies. That's really good. Frankly, something that OpenAI does not have in practice. You know, they came out shortly after actually our first report was launched last year, or sorry, just before with the thing saying, oh, hey, we have this internal whistleblower hotline. And then all these whistleblowers came forward without using the hotline. Tells you a lot about the level of confidence people over there have in their sincerity on that stuff. So anyway.

So just really, really good to see the emphasis there. So lots of stuff going on here. We'll see the longer version of the document. But as an initial draft, I mean, if you're going to put something together like this, I think this is a pretty good start, actually. Next story.

And one more story. And as always, we've got to have a story about chip bans and export restrictions. This time it's about TSMC restricting sales to China as a result of US export sanctions. So the rules are that TSMC is not allowing design firms to order chips made with 16 nanometer and below processes,

unless they use government-approved third-party packaging houses. So below 69 years, basically like top-of-the-line chips, even 69 years, I presume is like super old and nothing you want to use will be using that. So this is starting end of January, so already in action, and this will apply to NVIDIA, AMD, everyone works with TSMC.

Yeah. And the idea here is like, if you can imagine some Chinese company ships to TSMC, TSMC makes the dyes. So we talked about this in our hardware episode, basically the like the,

the GPU die, like the logic die, but then the logic die needs to be packaged with memory, usually from SK Hynix or Samsung or something. So those things can sometimes get packaged, not by TSMC, but by other companies. And so TSMC has to go, okay, I'm going to ship this logic die, which is kind of the crown jewel of the GPU in a sense, to this other country, maybe in a different jurisdiction that I can't control, where I just have to trust that the packaging

plant that's going to combine that chip now with the Samsung or the SK Hynix memory stack or whatever and package it together. I'm just going to trust that that packaging plant is going to follow whatever export control compliance stuff, know your customer or whatever.

the US government has asked me to follow. And in practice, that doesn't happen. It's a big vector for getting chips into places where they shouldn't be. And so what the US government is saying is, look, we need essentially government-approved packaging

facilities to be the only ones who receive dyes from TSMC that are 16 nanometers or below. It's consistent with some of the export control stuff that came out in the late Biden administration as well, which by the way, the Trump administration has kept in place. Kind of an interesting case of executive orders that have not yet at least been thrown out. So anyway-

Sort of interesting. It is the case that around as of 2023, China only contributed about 8% of TSMC's revenue. And so, you know, this is not maybe the big hit that it might sound like to TSMC having this extra constraint. Right. And actually, I'll throw in the bit of listener questions here, since it's quite relevant. We had someone on Discord asking for your take on the export controls on H20 chips.

So I don't know much about chips, but apparently H20s are good for reasoning models due to having more memory than H100s. And yeah, he just wonders on your take as to that general idea. It is changing. So new export controls come in that do cover the memory side as well. And that was, again, part of the last hurrah of the Biden administration there. So the H20s are no longer going to be shippable to China. And that's really important.

Because, yeah, they actually are, especially as we go into the inference paradigm, they are more important. It's also like you just want to be robust to whatever the new paradigm ends up being. So as long as you're shipping stuff where NVIDIA has room to optimize, where they're allowed to use their 5 nanometer, their 3 nanometer, like the TSMC 5 and 3 and 2 nanometer processes, yeah, you just don't want that in the PLA's hands. Like, why would you ship that stuff? Next story.

And we're almost done. We're going to do a quick lighting round for the last couple of stories. I'm just going to run through them in synthetic media and art. First up, we have Thomsen Reuters has won the first major AI copyright case in the US. This was a lawsuit from back in 2020. There was a firm named Ross Intelligence that was reproducing materials from a legal research firm related to them. They

lost the company Ross Intelligence. That company is now out of business anyway for a couple of years. But the important bit here is that the fair use clause, the kind of argument that there's fair use for being able to replicate a material of Thomson Reuters was rejected by the judge, which could be a

have implications for the ongoing lawsuits against OpenAI and actual generative AI companies, which was not. Next story.

And the last story is that Scarlett Johansson has called for a deepfake ban after an AI video has gone viral. So I don't know why, but Scarlett Johansson seems to be very involved in stories regarding synthetic media celebrities. In this case, there was a video posted on Instagram of various celebrities wearing a t-shirt and

with a logo of the middle finger raised with the Jewish Star of David and the word Kanye as a response to, I guess, Kanye West doing some anti-Semitic stuff. Well, that included Scarlett Johansson and other celebrities. And Johansson made a statement saying that, you know, she probably agrees with the idea. It is...

is not great that she has been depicted in a post that has gone viral on Instagram, slightly viral, and potentially could be misconstrued as a real thing that she was involved in, as opposed to something just made up by some random guy who wanted to make this post.

And we got to finish with that. Thank you for listening and thank you for all the comments on Discord. We haven't been able to get to too many on this episode. We got to finish up, but feel free to ask for a couple more topics or stories to discuss. We'll try to get to more of them next week.

Thank you for listening. Thank you for subscribing. Thank you for chatting on Discord. And more than anything, thank you for continuing to tune in as we apparently will never stop and keep going until AGI gets here.

from the labs to the streets

♪ ♪ ♪ ♪ ♪ ♪ ♪

From neural nets to robots, the headlines pop Data-driven dreams, they just don't stop Every breakthrough, every code unwritten, on the edge of change

With excitement we're smitten From machine learning marvels to coding kings Futures unfolding, see what it brings