#190 - AI scaling struggles, OpenAI Agents, Super Weights
01:37:21 Share

Circe chance.

This make. You feel hello and welcome .

to the last weekend I podcast we can never shot about what's going on with A I as usual in a bs will summary and discuss some last week's most interesting A I news. And as you can also go to last week, in that I have a text news later with even more articles that we will not be covering. I want to your host under a corner of my background is that I studied, they are at d and I now work at the genetic AI start .

up and on your other host jermy herri. Go under C O S, the ei national security AI company. And yeah, look more to travel than usual because i'm not particularly traveled.

The recent dad, that's i'm going .

to start to like to drop that every time I got sorry, sorry, you dad, I wonder how .

long working a hundred percent IT goes. It's nice because .

that makes the team like IT ever was. But yeah, I mean, I think the so so so the the thing that's on today is I just, uh on on let me go to do a thing at a bank um which which has a story about twenty minutes late, which means we going to try to end this on time, actually do a one and a half hour ish episode. I wonder how that's gonna we tell ourselves every time every time.

every time we'll give you a try. So actually moving on to a couple things. As usual, we will quickly acknowledge and listen comments.

There was a fun one on youtube I really liked. Uh, great podcast. Love the detail. Thank you for that. And then IT makes me sound smart in exact meetings, which is a great .

outcome on goal.

That's one of the goals for sure, is to make people sound smart in the daily conversation or an executive meetings. And then as I from that, I don't want to give a shot out to people who leave us ratings about leaving a review. We are now at two hundred thirty nine on apple podcast.

What number is scripting up? All is nice and we are at four point seven out of five. So hopefully that reflects us staying consistent in quality.

And now a quick preview of what we will talk about in this episode in tulsa apps, nothing huge sort of previews of what's coming more less in applications and business. A lot of stuff on hardware and data centers is the focus. Got a few exciting open source things, including alphago three, yes, a few pretty nerdy research stories and .

say Normal e fair.

Well, I think these are a little more conceptual. Let's say more about how they internals of these things work. And in policy and safety gonna talking a little bit about the E, U. And some conversations around nuclear, A, H strategy. The usual kinds of things, no major news there.

And one last thing before we start on the news, once again, we want to shut out to our sponsor, which is the generator rop psychologies into disciplinary A I lab focused on entrepreneurs ai and is a number one school foreign users in the U. S. For over thirty years now.

And last fall, professors from all across bobs and partners with students to launch this, uh thing generator, which is a lab organized to eight different groups, including things like air treated neurons and business innovation, af x society of future work in talent and saw. So this group has already pure trained a lot of a faculty of boston on a eye concepts and A I tools on the website. They say that we generated excEllence, entrepreneurship, PS innovation and creativity with a and they are fans of a podcast.

And we think for a sponsorship, you can go to the are website if you want to hear more about IT uh or just you know keep an eye for the news coming out of them and moving along, getting started with news. And in tusa, aps, we are actually going to start with a sort of follow up to something we discussed last week. Another article ask out in this kind of ongoing conversation around seeming slow down potentially in a ee development.

So this one is OpenAI. Google and anthropic are struggling to build more advanced A I from bloomberg. Nice overview article of the topic and last week we had discussed a little bit, but last episode, I should save as might come out, uh, little be closer.

But we discuss how the new model from open I or yan is seemingly not hitting desire performance goals, according to, in this case, two people familiar with the matter who spoke on a condition on anonymity. And this article also mentioned that people from inside google have said sort of similar things that the next generation of gina is not living up to international explications and also that, uh, inside on tropic there's been a chAllenge with three point five ops. And in all of these cases, they just referred to people familiar with the matter.

So in all of these cases, what seems to be happening is that you train these larger models. They perform Better on variations, but but not as much as would be expected. And this has that to a lot of conversation in the community.

You ve seen a couple people be like I told you. So everything is slow down. Gary mark is always uh, love to say us and Young and also kind of a posted something to that effect we discussed in a little bit last week.

And I think we can talk about IT a bit more might take is as before, that perhaps is not entirely surprising that the pure scaling approach of a bigger model, more data, more compute is chAllenging to keep going. And we know that uh, seemingly the scaling laws we haven't broken. So you're still getting the expected improvement in perplexity and being able to uh particularly probabilities of words or letters.

But in terms of hava translates to actual kind of intelligence performance on benched Marks are just being smarter, that's started to say. And in some sense, you know, even a quantity was scaling. Well, as we know of that, as you get Better and Better as you scale up more and more, you get something again to diminishing returns, it's harder and harder to get the same amount of improvement.

Uh, you need to keep uh, expanding to bigger and bigger sizes by orders of magnus de, roughly speaking. So we shouldn't be read. We take away, I think if we shouldn't be read as we are hitting a wall as like A I improvement is gonna low down necessarily. What this is implying is that IT might be at a point where it's chAllenging to keep improving at th Epace w e h ave b een b y j ust d oing s caling, which I think is not a tally surprising. And at some point, this seems like this would happened sooner or later.

Yeah, I think this is really interesting as A A conversation that i've been having with your friends the laps for the last year or so as they've started to see some of these things come out. You know the debate over what exactly is means for the future scaling. Um you one one observation is um you point me out the idea of diminishing returns, right and and that diminishing returns does play out at the level of these especially these log plots.

So you have for example, um the uh essentially the performance um you have to yet, like you said, keep exponential ally increasing the amount of compute x entity, increasing the amount of of data that you you feed in these models to continue the same H I mean the same linear AR friend performance. But but this is this is the problem, right? What does that actually mean for a model to get you know x percent Better auto complete? That's what we're talking about, right?

That's the thing that scales is like roughly how good this model is at predicting the next token, if you're talking about a little lam um how that translators the actual concrete performance at test we care about is the big question. In some sense, it's always been the question is just that, that mapping has historically been very tight. So as you see the model get Better, you know GPT one to two to three to four, you've consistently seen Better.

Next word prediction accuracy translate into greater um general capabilities, including even agented capabilities with uh GPT four right and and that was kind of insane, right? Like this idea that you get really going to text auto complete and at a certain point that gives you agented reasoning capabilities would have been insane just like a year before uh IT seems to be true. So one of the big questions is um as we do this, are we really building a world model that is more usefully robust, that allows for the building a Better agents um keeping in mind that we've now move beyond that, that just peer scaling eleven paradise we're now wrapping things up in obviously is post training, which includes now reinforcement learning specifically for um agent's behavior and we have now all these influence time scaling laws.

They give us a whole bunch more and they compound multiplicity vely with train time a scaling laws. I think that's kind of the right way to think about IT. So you can tap one out but not the other and you can still continue this trend.

So there's a lot of a uncertainty, I think, in in general. But when you talk to people, the frontier labs, um I don't think anyone is expecting a slowdown. In fact, i've heard quite the opposite one of the big themes as were seeing A I be used to automate A I research itself more and more kind of closing that final feedback loop that gets you to like fully automated research.

Um so yeah I mean, I think there's so much new ones here. Uh, difficult to unpack in one epsom. In fact, I think we could almost do we have a hardware episode? We have to do.

We almost do scaling law episode in the past, the kind of, uh, A S I. But I think, um yeah I I think fundamentally, the big blockers now starting to look industrial, right? The industrial base is struggling to keep up with the energy demands of this kind of scale.

It's hard defined that you know five hundred mego what uh sparc group capacity that you need for that next cluster the one giggle while cluster um you know five gig what seems out of reach. So so pretty soon, the scaling runs of twenty, twenty seven, twenty twenty eight are looking chAllenging to pull off and and will be talking about that as well today. But I think that, uh, you all these things exist together and not enough for just scaling to get you there because it's not economically viable from an energy standpoint.

exactly. So a lot of you here, right? And I think IT is worth highlighting that i've had the feeling over the last couple months that we are seeing with agencia.

I with agents, something is I came to what you scene with other part times of way. I like video generation, image generation, where we are seeing to take off. Yeah, I early days of K, I like we have in text image, like the text video.

We ve been a couple. strains. And at what has happened for last couple years is you've seen a few demonstrations of IT. And then in a year, in a few months, in a half a year, in the case of, uh, text to video, you know we we see more and more like there's kind of an of launch of the trend realizing itself. And that is is definitely gonna en.

Next year, we would end to the I we were going to see more and more with coming out as well we talk about in a bit. So in that sensor is not gonna and slow down, we're gonna new A I tools that can do even more even regardless. However, we get these largest models that are even smarter.

IT is a very interesting question whether we can also scale up with models and get a major improvement in performance. Uh, I think there's a real question there on, for instance, not just about this gave a model but maybe a data, right? We haven't used up over data persae, but we know that quality of data matters in addition to quantity of data.

And if maybe if that you sucked up over a news article, you sucker up all of a pedia, you sucked up all of step overflow, you know how much is there left of the good data, so to speak, right? Uh, so i'm sure we'll be seeing some research, hopefully coming out and expLoring the topics of israel potential for plato in the scaling laws because again, also worth mentioning, as we see o of an topic recently mentioned on the podcast, these are a miracle loss, right? So these are not so our physical loss is no theory behind them, to my knowledge. So we may, in practice, see some sort of plato and that the laws don't go infinity. And that will be interesting as well.

Yeah, I mean, my my intuition is that the laws persist because the sort of intuition behind seems pretty robust. I think this is all stuff that we need to talk about, that we we have to do an epsom on scaling laws.

And this is but though the one thing I all see on the agenda, ture, is the the last mile problem is right now, the thing that the people are working on that is the top is not to crack um for an off a lot of like long term reasoning task they're trying to solve with these systems, the ones often that deliver the most value. It's not good enough of an agent that gets the steps right ninety and nine percent of the time because you're going to have to string together no dozens and dozens of steps. So on average, you'll expect the thing go off the rails.

So that last mile problem, multi task coherence is so, so important and and that's the the thing is chAllenging to for right now because there are so few examples of long term reasoning traces that you can actually train on to speak to your under your example of a your matching of the data wall issue. Um there there are potential ways of around this uh synthetic data actually does look quite promising for this. You can have A I systems audit reasoning traces generate not IT generate not IT in a sort of um alphago style approach that's been been tried and being tried um and I think there is some promise there, but anyway, yeah I I think we need you a scaling epsom okay.

you heard first, do we have promised a IT and we will deliver at some point of next year .

yeah already. And next up.

a relate story coming up is that open eye is apparently nearing the launch of AI agent tools to animate tasks for users. So we not much to a story, not much, but we know yet. Apparently this new I agent is code named Operator, and we will be able to use a computer to take actions on the persons behind, very much similar to what unprofited launched recently on the API. This is according to people familiar with the matter and .

that was all over this.

I know the people familiar matter of real sources of news appear on the um so there was a staff meeting, uh, just last week and upon on the opening as leadership announced plans to release the tl in january as a research preview and through v API. So seems like IT is coming a relatively soon at least if these plans come true and won't be surprisingly given on tropic already launch this in their API when .

involuntarily pink by a agenticity m of the Operator model opening, employees are said to have responded, Operator, Operator, don't call me. I'll call you later. Sorry, that was exactly as unfunny as I said. Some real to get joke isn't .

IT isn't IT there's .

like three people that laugh yeah no, I think there's bunch of specular about what this model sorry about that everyone what this model might be like web brows or tool could be um your task automation of the usual stuff. I think right now um the best way to think of these is as just experiments in in long term test coherence. The problem that we just talked about any with that last mile thing, I think going to be the biggest chAllenge .

going ford to this and moving a long a couple of more stories. First up, google has dropped a new gm model and IT is performing pretty well. So there are of this germany dash exp dash eleven fourteen, which has stopped the alam arena, uh, the l marina chat, bott arena.

That's a new name, I think, for els that's swear I .

liked alm is Better, but stopped to chat by rena, where A M S compete in a head to head format and users vote on which ones they liked the best and so this new um experiment, gina, whatever IT is as much later stupid for o and I performed open a yes or one preview, although again, this is on which model people like Better, which is sometimes kind of hard to say how that really uh translates to intelligence and you know so it's now the longer the opening models, also greg two is up there as well. Yeah.

I M IT is at least a more direct measure of what human preferences and you know your standard scaling curve that shows you cross entropy or whatever. Um but but yeah mean IT is difficile to assess how much this means. They're speculation about whether this model is is this a version of ti one point five or whether it's an early glimpse of gi two to some degree. So often obviously version numbers are our kind of meaningless.

Um this is something that uh actually eo dario mentioned the difficult to the honest podcast like freeman of of kind of naming these models and and actually remember talking to a friend ah who told me like months and months and months ago is like, oh yeah so opening eyes training GPT five right now and like here's you here's some of the stuff some of the stuff that we know and then I turned out to be a different model like when I was released, the name is discharged. And so these sorts of things just happen all the time. I think a one, the thing this is getting out and I think the thing to anchor on is like, what is the computer cluster that was actually used to train these these models right?

When we're talking about G M I one point five, g nine nine two, the thing that really matters is is like is scaling working or or is the next training paradise working? Um it's possible to gm I choose a fundamentally different training paradigm, so that will be a very interesting distinction. So does these incremental can matter, but the fundamental thing that makes the matter is the size of the training cluster, the amount flops to go into wet or the the bells, whistles, ls that are added during training.

And and here, you know, it's very unclear what the answer to that is. So we will just step, sit back and and a weight really for more information. I do like the industry y's trend towards dropping these mystery models in the middle of like some some leaderboard, seeing people freak out. And then later you find out like sometimes you don't even know which company right happened with rock too. But uh, they are here.

Here you go. Next, we are going to talk about image to video. A new trend as well serves a shank shoot technology, and we've covered their ital video, I believe that already, which generate eight second clips from text.

And now they have an updated tool. You can give IT free, distinct images. So for instance, a shirt person and like a vehicle. And I can then create a video for you combining those images.

Uh so yeah, part of a trend that we've seen multiple tools that take an image and then create a video out of IT. Sometimes they kind of continue and and generator video that looks like what would be of the continuation of the image. Here is another example of a tool launching with something like bad capability.

And out to the last story, we have forged reasoning API bea from nub chat about news chat and hermis three in the last epsom. And they are also announcing this reasoning API, which allows you to query N A P. I like you do with an alarm.

But built into IT is a bunch of the known techniques for reasoning like research chain of code mixture agents at seta. IT is now available in beta to just a few users, and i've seen some examples of people trying get out IT does seem to empower kind of weaker models like lama free for instance. Wive's, uh, improved to leagues to do Better reasoning, uh, in a way again, showcasing with trend towards both reasoning and agents A I, which kind of go hand in hand.

Yeah it's actually like it's pretty wild looking at these metrics and I personally feel like I need to dive into this more um that they seem almost like, yeah just really absurdly effective. Thirdly, effective framework. So on her m three, the seventy billion prend version that for me three the model that they themself that news actually built themselves um they're clamming like jez um like eighty one point three percent on the math benchmark.

So that is a performing german I one point five pro um in GPT four obviously but uh as well as sound of three point like this is is pretty insane. And you see some of competitive numbers, maybe a little bit less perform at a competitive on G P Q A D M M M U prose. Well, the anne benchmark is the like weirdest one from what i'm seeing here like there.

I mean it's tough as a very small benchmark. They don't show here. There are on a ton of um of kind of samples in the in the benchmark um but it's outperforming nominally, even a one released. I want to look I love to have more information about specifically like the eval part of this story, but they are using money carload research um and h such chain of a chain of code like chain of thought specifically.

When you connect your reasoning trace to a code interpreter so you get actually grounded feedback uh that doesn't doesn't pollutant ate so everyone's a while you can to get that that clarity in your reasoning trace um and then they also have a way of of setting up of uh a query between multiple agents they call M O A or mixture agents. So it's this compound framework. Uh multicolor research is pretty probably part of what's going on in the o one set up is at least my guest for now though, honestly, no one really knows and IT could be anything but uh, these are those sort of intuitive things you might try if you're trying to replicate that.

IT seems superficially like they have. Um but again, I like I just want to see a lot more information about the about the actual side of things. Uh you know what what actually like on on the amy benchmark. I love to see you know the reasoning traces in all and qualitatively be really interesting to compare to the few uh o one reasoning traces that we actually do have because um yeah be be curious to see how that you .

know how those who compare yeah exactly there's a very few details in the blog post and industry with numbers really but secondly, I I think like believable that if you add and combine various techniques but that we all do you know can aid in reasoning in a good way.

You can then argument existing alleys and make them much Better reasoning, perhaps matching or one itself if you do IT well enough or a beast getting closer to you or on and outlook actions and business. First, we have opening, discussing an A I data center that would cost a hundred billion daughter dollars. So this is according to them having information.

Will the U. S. Government officials about these potential plants? Apparently, there will be five times larger than any data center that's currently being developed. So the, uh, opening eyes is top policy exec chrystal hain said at an event in washington that this company has shared information about the potential impact our data like this, and we really don't know too much more. We know what they are. Can talking to a federal government, they are calling on IT to expand the energy grid to be to enable these kinds of things to happen uh and they are suggesting various things like, uh, speeding up a permanent process for AI data centers like this yeah one of .

the things that they've called for in their breaths s is to set up A A national transmission highway acts typing to expand energy capacity. So basically just the national, whatever was like highway act, the fifty is right where we we set up all the the interstates in all that stuff. Basically we need that, but know for for energy frame cate is the argument here.

This is for a by the way, IT seems to be for a five gig what um cluster which would be probably the target cluster, right? The partnership with microsoft? No, I don't think they say explicitly, but that's what IT seems to be um you know like one of the things if you if you're looking at this through a kind of washington national security policy lands like great, like good shit, we definitely want this in the united states.

But um you can probably think about trying the a massive package like this to require ments on the part of the labs that uses infrastructure to hear a certain security standards, right? So there are whole bunch of really i'm saying this because because we're working on an investment right now like lab security and all that self. Um there's basic stuff that I think opening eyes is like keen to rush this through and be like, yeah i'll give us goodies for free basically or you know well somewhat for free, i'll let them spill over happen and academic institutions will be able access computer self like that.

But fundamentally, um just go and do the thing. I think this seems to be tight, like some pretty intense conditions if were starting to think of of AI as a national security technology like the security situation, is the business more right now and kind needs to be need to be fixed up? So for a hundred billion dollar compute cluster I think, uh yeah five giga watts this is yeah it's a behavior like there is not five giggles ts of spare capacity anywhere in the grid.

Um you know every company i've talked to is like yeah that you right now were at most we're thinking about one gig. A lot realistically were in the fifty a few hundred megawatt range for stuff is up and coming. So somehow that's got a change if we're going to be competitive with A C C P. And the the best way to do that is obviously a big infrastructure build up. The question is just how do you time incentives to .

to use of that infrastructure? And speaking of data centres, the next story has to do with another one from X A I. In this case, apparently, uh, X A I has gotten an approval to use a hundred and fifty megawatts of power, which would enable all a hundred thousand gp use in that massive AI data centers to run concurrently.

Apparently, so far, we've had an initial supply of eight megawatts, which is not sufficient to actually run their thing IT IT seems like IT would be an estimated model of hundreds and fifty five megas to run all of those nine uh one hundred gp s concurrently. And so here of this approval IT seems that we are making progress towards being able to use all of that GPU power, which um you is is still they're still a lot of uh infrastructure considerations there. That may not be the case that you would even want to run all hundred thousand G P S. concurrently. But certainly now they have the option.

This is the the big X A I colosse a cluster is really like gia factory of compute sometimes called and yeah the and it's also the one that you're Jenny wong um had been talking about in terms of the the insane speed of the build. I guess he was like nineteen days from you are rolling out the first uh the first pause onto the the floor of the data center to you know doing a training runner, whatever and um ridiculously, ridiculously fast that i've seen stories, by the way, of like competitors flying planes like like sessions or whatever over the site just to understand how the health on pulled this off.

That's like the level of freak out that we're out right now in the in the space and um what you want did by the ways like because initially there was only eight megawatts available of energy on the the time the opening of the day center july, he set up paddle power generators to kind of bridges the gap. And there are whole bunch of really interesting tweet. If you're into the hardware side check like check IT out.

I thought they were really cool. Uh, of these like new X A I employees come in and saying, yeah you know he brought these in from tesla and use them to bridge the the energy gap there um in right now. So the big question is you know how how fark in this go? There are plans to double the capacity beyond one hundred and fifty mega watts um which would know further increase uh the G P U capacity.

Hundred fifty is enough for one hundred thousand um uh eight one hundred and um and so anyway, uh this is going to be a really, really big facility. Obviously currently he's referring to IT as the largest cluster in the world. Um IT wasn't the largest fully powered cluster in the world. Um but but it's it's now truly gonna uh on its .

way to that next story also on a hardware but chips, not that the centers we have new results from the ml perf benchmark dealing with and video and google ships. So the latest generation of chips and video case that we be two hundred GPU and google case that the trillium accelerator and both now have kind of more clear results to beyond just what the companies are saying.

So in google there, six generation T P U trillium has shown a free point aid, uh, performance improvement about four times Better. And the be two hundred GPU seems to be about twice Better over the age one hundred. So in both cases, seems we're not hitting of the kind of ceiling and improving ship performance. We are still doubling or even could rule ling in performance with each generation of chips ah there .

is also a more energy efficiency, especially with the the google chips, apparently sixty seven percent increase in energy efficiency when you're talking about know the difficult thing of hitting that you know give walk, cluster and so on, like that's A A really big factor increasingly. Um you know people often think about energy through the lens of yo climate. Not that up, but the thing that really matters is how much can we can we freeze out of the the grid power that we have um increasingly, that's the bottle neck. And so really important new google starting look very president setting up their TPU program like I don't know, I like I even remember .

twenty sixteen.

twenty fifteen and yeah I mean that was like back and sort of wild and now you're worrying to T P U V five or on the you know trillion and all that stuff um and really paying off.

One of the things that google is especially good at is the multidrug center computer game plan, like basically you and is relevant right when you run out of power because if you don't have enough grid power, that is enough grid power in one location because that's the real chAllenge. Then the question is like, alright, well, I guess i'm going to have to set up data centers in different geographic locations and have them work together. And that means I need to be able to do distributed training across like a large set of data centres.

Um that's something that google had cracked early and hard and you've got create papers. We've talked about some of them on the podcast. Um but but don't sleep on google on that.

They're really good at design. They're really data center design. These are not like crazy. Geographically separated, by the way, they're in the same general area for for technical reasons ah there are a little bit chAllenging and overcome right now, but but still, uh this is a big competitive vantage.

The other piece on scaling, so the new TPU, the trillium, can link up to two hundred and fifty six chips in a single pot in a single hybrid with pot. So we're talking about high band with, again, this. I'll be punted to a harder episode.

But the GPU to GPU communications, the things that like on invidia devices, would be done over like N V link. So N V link is like the yeah G P 的 G P U super super high um high capacity uh cables。 So he usually you'll see like I don't know, thirty six, seventy two.

GPU in a pod this two hundred fifty six and um and so so really, really expand ables well beyond that. So very much designed to be super, super scalable ah and they got all kinds of instruction cure for IT. So yeah this I think google is want to really watch there there the they were the sleeping dragon that got woken up by ChatGPT. You know, now they have actually quite a strong computer vantage over microsoft and .

open eye and moving a away from a hardware bit. We have almost like a gossip story would call this. So we are getting a bit of a detail on the uh, recruitment by me, ama adi for her new venture. So is V C T O. Former C T O of open I H SHE left pretty recently and has announced she's doing something we don't know at all what this new company would be, but now we do know that a fair number of people from opening I are joining on endear. So here we know that me on a chin, a research program and manager, is during this new company, apparently also x head of post training barf and former senior researcher luke mens, uh, who also left open the eye on recent months, are teaming up on this so no details beyond that but seems like a yeah a lot of talent being a uh combined to this new company.

Yeah I mean, deal the the one thing you can you can pretty safely bet is that this is going to be a long asia. I play with this team. Barret office is one of the original uh, co inventories of the mixture of experts model.

So uh, you know very good at the know on the foundation side and retraining is presumed ly, i'm going to be part of the game plan as well. So they have the productivity side in miana chen, and they've got the more kind of pre training um obviously like barzil actually was was ahead of post training previously open eye but certain ly has the the pedigree to do to do retraining stuff. So does look mets music to a senior researcher or so. Anyway, it'll be interesting to see I think IT maybe it's just another another A G I play. I would not be a little surprise if IT wasn't actually .

and one more story dealing with business. We are seeing a little more rumors about on topic as their quest to get more money. So proud the amazon is discussing a new multibillion dollar investment in our topic.

Uh, there's apparently discussions on something similar to the initial four billion dollars amazon invested in entropic a previously last year. IT seems at this time though there might be uh cava debt where on topic would need to use amazon chips specifically to train their models and these chips aren't in video. So that could pose and ical charges these training num chips apparently so h again, not too much known this is just internal discussions but um could be interesting to see if this does come true yeah .

and apparently like this, according the article, they say that any investment deal with amazon could come in the form of convertible notes that become equity after anthropic raises capital from other investors. This is a little weird.

Um usually this is something that you see in early stage startups, uh, when you're trying to avoid having to set evaluation for a deal, right? So essentially we are doing is you're delaying the discussion of what the valuation will be until there is a Price ground later. This usually happens earlier stage because it's really hard to value is start up early on. You just have a couple founders in an idea.

Um so yeah not sure like exactly why the structure the deal takes that form sir reminds me that the safe that H Y commentor that you were that anyway Angel investors use a lot um but in any case, I think this is a really interesting deal to watch the the a strings attached obviously are highly strategy for amazon trying to force anthropic to use their hardware. There are also they're alternative to kuta too. That's going to be really important because emmy's on has an awful lot of catching up to do, right? They were called flat footed.

Um there's no other way to put IT on the whole agi scaling race. And now they trying to catch up, you know they're not developing their own models, are really leaning a lot on on anthropic to provide those another turn of four santhal pic to generate demand that can help drive the improvement of their own hardware. I think this is a really interesting one to watch, to understand how stable, long term stable, the relationships between hyper scales and model developers like the opening ye microsoft relationship Candy in in the long run, right? Microsoft of open ee were already seen cracks in that relationship.

Um to the extent of anthropic is now being sorry at the amazon is now being kind of forced by business by business pressures to uh require anthropic to ask anthropic to use their stuff more and more to the point where maybe I won't be workable. Uh the kind of makes you wonder I about a lot of these deal. So um yeah, this is maybe a canadian a coal mine.

But IT has been a very fruitful partnership for the two. We don't know the details. Um I think the big question is just gonna like what specifically are the requirements one of the strings attached? How much training um .

training um .

training um training come on guys has to go into this .

onto projects and open source. We got a couple stories here starting off with the exciting news that alpha fold free has been open source. So we got the least of the source code and model weights of this is meant for academic use.

So the license is a bit more restrict tive. And uh, there's really not too much else to say. We covered of fold free previous ly, obviously, this was a major improvement in the ability to model proteins and in be able to apply IT for things like scientists, discovery, drug development. IT was not open source when I was announced ah and sort of a seems like kind of body know where this suddenly came out yeah I think the law in the .

eyes the third indication of your model is the one that you have to like withhold right that's how GPT three happened in half full three um well anyway no longer I guess. But ah they have a deal where you actually you can access the motivates if you have google explicit permission for academic cuse only. So and they may be giving themselves a little bit of google room there to the partnership with deep mind and isom.

Thc labs is really where a lot of this stuff is being monotoned on the I I don't want call google and i'm i'm not actually sure exactly how the ownership structure works there, but they are kind of partner organizations in some sense. Demis actually helps uh I movie labs a lot as I think it's executive um but yeah so so you know theyve got A A diffusion based approach anyway the whole thing about this is it's not just about model in the proteins. It's about modeling the interactions between proteins when they're modified and legs and thinks that find the proteins um which which is actually very interesting right when you think about medical impact, right you're often concerned about how will these two things actually interact that the only way you can make an effect up in the human body.

And so this is really where qualitatively alpha three is is head chillers above alfa two. Um so be interesting to see if if there's actual tangible impact that comes from this there there husband, some from health IT fold two. I think it's to say a bit less a bit less than people expected when I first came out. So we'll see if that changes here.

Yeah and yeah, to get access to the weight you should need to fill out, uh, google form. And I like A A little self questions and deep mind will then decide where to give the weights to. So everything pretty K G preak.

careful. And they really emphasize that this is not for commercial use. If you university non profit organization research institute, you can use this is. But they highlight over and over uh, that this is not commercial use. And in fact, there is that some sort of like a step, some sort of marker and the weights related to reform submission, which is interested uh but certainly good news for scientists for researchers and and I think they did shares uh PVC already in the sort of closed off process services, expanding uh the access state and anyone can look to the source code for inference. So either way, even motivates of this can and inform people as to how to build resource of models.

So does that apply their water marking the actual like like, uh protein seat, like a new essence. Cy is something that are generated by the thing I am not like.

Each alpher fold free model promoters file will contain a unique identify specific to VS form a submission interesting OK, yeah. And next story IT is that near plans to build the world's largest to one point four trillion from open source A I model. This is a near protocol and this is just a plan that they are aiming to kick off.

So they are aiming to crowd source this uh, training. They want to have thousands of contributors. And for now, you are able to start contributing to the training of what they say is a small five hundred million parameter model when they're kicking this off today.

So very hard to say if this will come to be one por trillion pounders model is like free and half is bigger than meters. Biggest lama model. Very, very chAllenging to train personally and a little, but we will be cool to see if you can even get to a fraction of this.

Yeah so so this is actually is kind of interesting that the two guys who are compounding this company or or former open eye guys there that actually part of the the transformer research work, the attention is all you need to follow up work that to let to ChatGPT. And that's interesting uh, because this is actually the first time i've seen a kind of A I mets crypto project. I am there's a potential um there's like bit tensor as well. I was a bit tensor.

Yeah I think there has been many initiatives on crypto in front, but we even talked about. So not sure how meaningful has .

been so far. I mean, so bit tensor was the first time I remember one like okay like this could actually this could actually work an awful a lot of IT to me to me and I just just my bias but sounds a lot like you have been girl just getting really excited about you know singularity that's latest total like it's like almost it's not quite it's Better than word association but like some of IT is actually just word association.

This is the time was like, you know what this actually makes some some sense so um the pitches right and this is not investment advice. Uh the pitch is something like um IT IT takes a lot of A T capital to uh to train at a big model right in this case the biggest one, the contempt training would be one point for trillion grand me IT would be a hundred sixteen million dollars to train and so on. Um so so why don't we uh create a new token, a new token for each model we want to train? And what we'll do is will auction with that token to raise money for the training run.

And then if you all those tokens, you can use them leader to buy cheaper inference. Or should I say you can use tokens s to buy tokens anyway? Ah so the point is that actually kind of make structural sense like this is not the crazy should i've ever heard come out of the crypto s space.

And the other interesting thing is got a little good like kind of glimpse to the the technical details. They say um you are to do decentralized network to have this like decentralized network of compute, which is what they need to, to pull this off. You're not going to have tens of thousands of gp s cried into one place.

You're gonna need as they put in a new technology that doesn't exist today because all the distributed train techniques we have require very fast interconnect. Okay, that's true. And but he added that emerging research from deep mind suggests that possible. Now when we talk, we think about, okay, deep mind and distributed compute in this way. The first thing comes to mind to me is the logo.

Um we've talked about that quite a bit a turn membertou is new serum, another company that uh this is an open source kind of version of this rely on that infrastructure if we're getting Better and Better at this sort of district of training um doing IT for really large scale training runs um even across like getting multiple data centres, multiple clusters to work the other really, really hard. Um so you know I think the local of that part of the plan here is only going to be part of the plan. There has to be some other solution ery that goes on. But anyway, I flag IT because I thought, you know, it's pretty wild sounding story, but it's actually just sensical enough that I like, you know what, this is not the crazy I should i've ever heard uh, when IT comes to using crypto, like the intersection of cypher N A I. So so there you .

moving on to research and advancements. We've got a super paper to kick things off, titled the super weight and large language models. So this is one of the type of papers that dives into the inner workings of large language models.

And discover is something this case, I think, very interesting. Uh, this is a partnership between the university of not her dum and apple. And what they say newspaper is that certain weights in LLM, what they call super waites, are kind of massively important.

So we know that weight, the parameters of annual net, very important. You can, you can set a bunch, jump to zero. And IT doesn't really affect your form performance. And that, in fact, how people can scale down models, make them more efficient, impress them. A lot of IT is by finding weight that don't matter, and then killing goes off.

And what they show newspaper is that, you know, we know that some weight are important and you don't wanna kill him off, but in in fact, they are like these super weight that are even more important that if you just zero out this one weight, literally, that needs to a massive drop in performance. And that's like if you just remove seven thousand of the other largest weights, we weight that you know contribute to activations when you on that, that isn't as important as this one weight. So you can say this one is like more important than thousands of other weights that also matter.

And what I found interesting here is this builds on previous research from earlier this year I actually was not aware of, and I think we talk about there was the paper titled massive activations in largely language models that had already demonstrated that there are these master accurate in newspaper of column supermarket activations that again, are these uh outputs in the internals of a large language model. So activation is just like the output at a given place in a your net. And so there it's been shown this past year, really like months ago, that you heavy and IT seems that we super wades play into that.

Uh, so lots of kind of interesting ideas here. I've kind of surprising to find this out that there are these special weights that are massively important. Don't think i've seen anything that hinted at that so far.

Yeah, IT is genuinely fascinating. I think mechanically, it's really interesting too, right the way. So these weight, it's not that they they have really large values and that's what these are, the larger activation or whatever they can take a arrange of values the way they're found is essentially by like looking at um essentially the the all the layers in the in the transformer and specifically the mlp layers.

So let me take a step back here to say when you have a transformer, right, the transformer is made of of blocks and the blocks are all stack together. And then each block has a two different kinds of layers. You get like a self attention layer, usually the the winning and an an mlp layer, which is basically just like a just a vanilla ural network that follows.

And kind of massage is the data that comes out of the self attention layer. And if you zoom in on that mlp layer, that second layer there, there are two steps in a couple different steps in that layer. The first is taking up the kind of low ish dimensional outputs of the attention layer and mapping them up to higher dimension.

Kay, so, so you move on. Maybe I like five, five hundred twelve dimensional layer amplified to two thousand dimensions, say. And then you can kind of in that higher dimensional space on massage.

In a way, this is sort of like allowing, you know, your papers to spread IT on your desk more so you can work on them Better. And then you do a down projection to recompression back to the original dimensionality. And what they found was that super weight consistently appear in the downed projection part of these M, L, P S, so that they keep showing up.

They keep showing up in the the blocks mlp layer, and specifically in the part of that layer that takes you, not that doesn't blow up the dimensionality, but that compresses the dimensionality after the number of mixing has happened. And so it's sort of interesting you know why that exactly happens. A super weight again, IT isn't necessarily largest weight by magnet de even at that layer.

Um and another kind of indication is to how this is working. These tend to happen very early on. So in very really kind of early transformer blocks of the model, what they'll do is h, they'll actually kind of look for really high activations that persist through the all the the the layers of the model.

And once they find the first instance where that happens, they'll kind of traced IT back and try to mess with the weight until they make you disappear, thereby identifying this super weight um the effective way that this the way that this happens in practice. Again, the mechanistic side is uh the super weight seem to suppress stop ward probabilities. So there are certain stop boards um like like the like the period character or a comic character that will cause the the model to you have stop generating outputs.

And um anyway so the super wait seems to essentially suppress those storms caused the model to keep generating outputs. And if you knock IT out, not only do you see that change, you actually you know see the quality of output. Just go to shit like you'll see some the output that they they show within without the super weight and it's like it's from beautiful coherent text to complete garbage.

And um anyway, I thought thought that was really interesting. The super high activation is super activation that you get the first time. The super way no takes its effect is actually persistent to across the layers of the transformer.

And so you end up seeing IT persist through if you tracking at high level detail through skip connections. But IT IT keeps appearing over and over in this kind of robust way. And so I think there's a lot to chew on from a mechanistic controllability same point on exactly how and why this is working. Um but but like you wondered, like I was not tracking the the first uh super activation results. So reserve it's nice because we're able to get a little bit more of an explanation now at the same time in and this is research as well coming out of apple too, which you know not not Normally to me, known as a big international .

ability powerhouse. That's right. They do break IT down a little bit. So IT seems that the two are not exactly identical. If you restore a superannuation but still of weight, that doesn't a kind of reverse for effect. You still lose a lot of quality.

And just to give a sense of the impact, the highlighting example where they say summer is hot, winter is x and so you know a Normal and will say winter is cold verses if you remove IT IT would say winter is a uh then this is probabilities. The court will have a high probability moral. And then the probably would be most build out and you're just save.

So not too much more information or interpretation of what this is. What we also doing a paper is a kind of on a more practical side of things, uh, explain how if you do a super outlier away, partizan, as they call IT. So if you quite when you quantize, when you lower resolution of weights that affects your activations across when you on that and it's already been shown that, that you can do Better quantization by kind of keeping an eye out for important weights. So in this case, what they show is that if you, uh, are careful to preserve these particular activations, if particlar weights, that leads to a much uh, less drop off in performance. So also from a practical perspective, IT is very useful to know for being able to uh reduce the size of models while not solution.

Yeah it's actually pretty like they show you can take the like all the weight in the model and and do a big quantization, but then you restore just the original super activation value in in like sixteen bit.

And so just keep you will keep the high resolution for that one and you'll you'll recover an awful lot of the performance that you lost from we would have lost from the aid quantization of the whole model, which like I don't to meet super counter intuitive to to my mind and i'll close with this. Um the weird figure in the paper by far. And this is like again, things I never would have bit on a million years.

There's figure six. Um what they show is if you take the super weight. And you scale up its value for some amount of scaling up.

For some amount of scaling up, you will actually see the zero shot performance of this model on certain tasks consistently go up. So across a bunch of different model sizes, they tried this and they consistently found there are some scaling where you get improved quality. Like basically just like take this way that already been trained increases value and that's consistently of a good move.

That that makes no sense to me. That seems weird. Um I mean, maybe you could argue that that I guess through regular ization or something that I I have to think more about there, the training scheme.

But maybe there's some like come regular zing pressure that artificially dampens the weight relative to what IT you might have been otherwise is not elastic enough for something. But but still I I find this like fascinating and I wouldn't expected IT. So a lot of weird, quirky results here that I think you know that the mechanistic interpret ability people should take a look at because pretty cool.

And out to the next paper, also dealing with understanding how large models work, in this case, diffusion models that generate images, and the title of paper is compositional abilities emerge, multiplicity vely, expLoring the fusion model, honest synthetic task, so they hear, explore, composing things. If your model can, let's say, create a square and create a rectangle, the image can IT then do something like, I don't know, I try and go on top of a square to keep to give a simple example of all, you can, of course, uh, you come up for many examples of the sort.

And the question is, how do these abilities emerge? How can you compose different concepts in your outputs? And the details are a little bit nuances, but at a high level, they say that the ability to generate samples from a given concept emerges depending on the actual data and the process by which you generate data.

And there is a sudden emergence on the ability to compose tasks, a to do well on test that require compositionally. So to be honest, we haven't dug in deep enough to explain this in a ton of new ones jeramy. I think you can probably do your deep dive explanation and do a Better job on this one.

No, no, I mean that's great. Like so the first of all, I think this paper from security of a system um A I risk national charities same point is really, really interesting. So has been this debate about the sudden emergence of capabilities is in language models where, you know, you train, you train, you train.

And then out of nowhere, seemingly suddenly, this model can, I don't know, help you, like, make progress on designing bioweapons. IT can write nowhere. Like, where do that come from? We have predicted this right. And a natural um model for the appearance of those capabilities is to say, okay, well let's say like designing malware require skills X, Y, Z right um and and let's say that over to the course of training, the model gradually becomes Better x, gradually becomes Better y, but gradually becomes Better z and but but to do the overall task right, you have to do all those three things together.

If you have eighty percent performance at x, seventy percent at y, ninety percent at z then you're overall performance probably is going to be something like eighty percent times seventy percent times ninety percent because you have to succeed at each one of those together. In other words, multiple play timely to perform the dangerous capability properly. That's kind of the threat model.

Um I don't know if that's literally like I did. I don't think it's national security motivated, but I think that's probably the biggest implication um for national security here. And and so yeah basically what they they do is they make this really concrete.

They take a division model, so basically an image generation model and they get IT to like generate different shapes, colors and sizes of objects, right? So you think like blue, small blue sphere, right, something like that. And they check like, okay, how performance is this at capturing the shape, the color and the size um and what we'll do is there they will specifically not train the model on certain combinations of those features.

So make sure that the never the model never trained to make a large purple square, right your large purple cube, and never never trying to do that. So you trained on a bunch of other combinations. And then you check to see how well can you perform at that, that new out of distribution test that I was never trained on. And IT turns out that its success rate at that task is essentially multiplication based on its success rate at the individual components, the shape, the color on the size that went into IT. They compare their uh sort of multiplication model to an additive model that doesn't perform nearly as well.

At explaining the the emergence these capabilities and yeah I mean and this kind of make sense look mathematically um if you've got a backroom engineering anything like that uh physics you probably know about the direct delta function um how basically in the limit, if you have a large number of change have to happen together with with probabilities lesson one um what you'll find is like the the the output your success that is basically gonna zero almost all the time because you only have to feel at one of those things, only have to get zero one of those things for the overall success rate to be zero. But there comes a point where all the sudden you can crack the last nut, right? You you're like, you're a sixty percent on one thirty thirty, all the stuff.

And there is. But there's one thing that's really holding you back in all the sudden you cracked. And then I have nowhere IT almost seems as if, in retrospect, you uncovered this, wow, this amazing new capability, when, in reality, that capability was actually a compound of many sub capabilities, each of which have to be chained together.

And when you multiple a lot of numbers together, you tend to get zero. If those numbers are less than one, that is, or between zero and one, you will tend to get zero uh on lesson until that all those numbers kind of hit a minimum threshold p and um anyway that's what this is all about. It's in some sense and obvious consequence of the same kind of map behind like the ah the function um um uh uh I I I think is really interesting for that reason but cool paper for for A I .

security yeah and from a practical perspective. So most of the results here on are on this synthetic tasks where they specifically do like shape, color, size, things like that and they show that in that pacing c case, you do see the emergence of the combination of these concepts and ideal a lot with the idea of these distinct concepts and how well you do uh in them.

But they also do have uh more practical tasks where they a look at celeb a uh where it's it's like a bunch of faces. They are attributes like gender uh expression, like smiling and hair color. And in this a example, you can actually look at the concepts, look at the performance on are things like, you know how will you doing at, uh, faces of men and women and you see something similar to that synthetic case. So in fact, this could also help in practice to mitigate biased outputs in emergencies, just as one example.

Yeah, I actually one last thought too. And I think this is actually pretty important for for large scale trains and A I this whole debate about emergence, right? The second emergence of capabilities. They're been a lot of papers back and forth. I think somewhat potentially you've had people argue like, oh uh, emergency in a real thing because in retrospect, CT, we can find that this capability actually did smoothly start to emerge over time. But all that stuff is like, if you know the right metric to look for in retrospect, in practice, we get surprised.

We train new model and we go, oh, shit, we didn't expect you to be able to do autonomous cyber attacks and IT just is um yes, in retrospect, you can design a cyber autonomy benchmark that will retroactively explain how you got here, but you need to design the exact right benchmark to list IT those capabilities um if this now helps to focus what that debate actually was all about, fundamentally, that debate had been about identifying the right set of capabilities that compound together to a lead to the capability you're interested in. So the reason that emergence is an issue in practice is that in practice, we actually don't know the full set of capabilities that need to be strong together to give us a certain kind of overall dangerous capability or or interesting capability. And so I think it's really interesting through that lens.

I was a new language or a new way of thinking about that. I was going to cut that agal debate. I debut from the two, three years. So maybe that.

and a couple more stories. The next one is titled the mixture of transformers as parts and scalable architecture for multi model foundation models. So another mixture kind of approach here.

We've have mixture of depth, uh, mixture of experts. And this time it's another one of those kinds of things. So make sure of experts we know is a pretty major deal.

The idea where is basically that you take an input and you have a router that activate certain weights for certain types of tokens or inputs. So that means of that you can use subset of the entire network for certain kinds of things. And as a result, you use less overall computation while generally getting Better performance by training more overall weights that are specialized.

And one way you reply this is when you're dealing with multiple modalities where you can have different modalities being routed to different experts. And this paper is presenting a sort of kind of a journalizing or uh, a very specific version of that for multiple modalities. So the idea of mixture of transformers is that when you want multiple modes like images, text and audio, what you do is you have quite literally separate transformers.

And when you have your input, you group with different modalities. You uh have attention over all those modalities in the sequence. But a new route, each mode to its own little al transformer.

So you literally can split them up across different weights. And what the show is by doing this, of course, you get the benefits of mixture experts by um doing overall less computation. Uh so you mean of those middle layers, uh you have typically feed forward layers, things I got that need to get the entire sequence.

Uh, so by treating each modality separately, the individual transformers in that mix can be smaller and have fewer weights. So we evaluate this idea verses a more traditional sort of mixture experts and across a dance transformer and the show uh uh major speed up uh doubling of speed in the the training while getting uh also good performance. So uh yeah it's it's really building on to explorations and how you do multi modality was kind of two predominant approaches and vis. Extending one of those types of approaches yeah there .

is a lot of the interesting potential advantages to this approach um and and we do so they have distinct they read distinct ment out medals to distinct um no say experts, spratly transformers um but then they do have global attention that that spends across them. So you benefit from across pollination between modalities, right? If there's one token that's um you know like you interpreted visually, say in another that's interpreted text wise, um you can still benefit from from that h interaction which is really important because often there is kind of emergent information that comes from, say, the description of an image couple to the image.

The other adventure ge they have is so training stability, right? One of the big, big chAllenges right now is you scale up these models enough and pretty quickly uh you run into issues with with training stability, uh you know get getting a loss to kind of stability uh drop over time and this model so mixture of transformers um because IT uses simple modality based routing IT avoid the additional complexity of having to learn the routing process in a mixture of experts, which is one of the things that makes training particularly unstable. And so so essentially this is really helpful for scalability. Um they actually show that so they show that A M oes in their experiments um get some diminishing returns at above a certain scale um and um and and above seven million meters and scale in their experiments more more notably that stuff that I am you know you you can come up with the engineering solutions for a lot of self, but the bottom line is you know mixture transformers is easier for them to use for these kind of the larger end of that scaling spectrum. So maybe an interesting option as well.

And lows paper is contextualized evaluations, taking the guest work out of large a language model evaluations or a language model evaluations. Video here is that in some evaluations you have tasks or inputs that may have different valid a outputs, depending on the context. So one example they give is, princess, what is a transformer? If your context is your an electrical engineer that has a different answer than from if you are a machine learning engineer h as an example so they tackle that general topic and save that if you actually specifically provide the contextual good of you know, why are you asking this question? What kind of background are you coming from a seta that can lead to more reliable and kind of meaningful ah outputs?

Yeah I think it's also quite interesting as a colour of the the issues that we have with model valuations right now. They found that if you so if you just increase that context, the ran experiments with this, but you can actually flip win rates between model pairs. So you might find, for example, that I know jit, like gami one point five pro seems to outperform claude three point five on IT along a particular benchmark.

But when you provide the additional context, all the sun that can flip, right? And so is a question like which models work best with the unity and which work best with with more more context about the user who's asking the question. And part of that is just like what's the baseline behavior of the model, right?

If the model is more baseline skew towards um responding to you as if you're five years old because of the way was pre trained or fine tune, then then it's just going to be Better. And if you have a benchmark that is geared in that direction to you'll find the model perform is Better as well. Um so I thought this is kind of interesting as as a bit of flag like some of our are relative rankings.

IT even makes you think about, for example, like alams is leader boards and things like that a little differently, right? Because now you're thinking, well, okay, sure. On average, people give a higher ello score to one model or another, but everybody is different. And so some models may actually just be Better at meeting the needs, the explanatory needs of one kind of person, uh, rather than another. The the upshot is you want as much as possible evaluations to factor in context and they do find just more reliable, more robust evaluations come from doing that um to prove that they actually can create A A sthetic dataset of of query. So we'll take the the basic queries that they get from some standard um kind of Q A benchmark and then they'll just like automatically generate a bunch of context about who's asking the question and then they'll feed that to the model and evaluate based on that context that results in a much more robust stead of responses that that presume ly are Better evaluation of the models. Actual capability to adapt to the needs of particular user.

And moving on to policy and safety with her story is dealing with the E, U. The tidal is the code of practice for general purpose. A, I offers a unique opportunity for V, E, U, and this is an opinion peace converted by new ia. Oliver. And you show a bangle you sha benji being a major A I researcher who is also a major proponent for A I safety.

So which gist our article is that are just recently on a number of tees, where was release of the first draft of the code of practice for general purpose ai? And this kind of practice is answering question for general purpose AI, which is things like ChatGPT. How can you build these models while also addressing the potential risks they create? And this a article is basically making the case for that code of conduct that sort of addresses uh the potential uh, criticism.

So IT says there are many stereotypes about europe's approach to regulation and then IT calls out, although many do criticize with you for over regulation, for slowing down innovation, things like that, they argue that the code can create a flexible, targeted and effective framework for ensuring, driving G P A I geno purpose, a innovation while respecting the rule of law. And we shared rights and values of V. E.

U. And they say that this has global significance, is this is the first time that legal rules are turned into more detailed guidelines for responsible development and the employment of general purpose. A I so not surprising, I suppose, that there was a sort of code that was published a long side. The um actual law of the E Q A I act go to practices is kind of suggesting how company should uh act to a follow the law of the u regarding I need general to develop models in a responsible way.

Sort of funny on on ex about people talking about how the sometimes that feels like the the E U A I legislation regulation complex is just constantly teeing up the next policy document and one policy document just seems to bleed to the next almost effortlessly and this definite has that feel um nothing too surprising, I guess in the set of um of recommendations that um that bengie uh is is calling for in this document.

It's stuff we have seen him advocate for before. Element with eu rights and values, element with the A I act and international approaches, proportionality to risks. This obviously is the big question right like you can say, okay, where to do this in the way it's proportional to risks, but what specific complaints measures what specific risks that almost the entire debate, right? That's that's where where people disagree. So um they also calls for a future proof approach. You obviously we've seen um I I think you know people talking about um this of like tear based, compute based approach to regulating A I systems .

are I think have come .

out looking pretty good, especially in the wake of the inference time scaling laws or this idea that um regulation through the channel of compute is a really promising approach because there's no other way really in the future proof what you're doing.

Um the one trend in A I is if you want to do something really powerful that could have eventually dangerous capabilities, you're gonna go through a regime where do you have to scale your computer in a really intense way? Compute is expensive. It's easy to audit.

Um it's got all these features that make IT a natural focal point for regulation much more so than models or applications which are incredibly difficult, like if you wanted to regulate at the application level, which a lot of people have been pushing for, um you know you'd have to get all of in people's business in a much more intimate way like the end users business instead of the model developers business where the more developer has far more resources to comply. So anyway, I think this is something that that is going to be the topic of a lot of debate. A lot of these things are like directional things that already already agrees on the question. Again, this can be how how do you actually incentives .

this and the next story dealing with safety is from a propionic. The title is free sketches of a cell for safety case components. So A S, L is A I. Safety level is a part of the responsible scaling policy, or speed of unpropitious. And the responsible scaling policy outlined exactly what these city levels are.

One food free category is kind of v potential risks that come with models at different levels of capabilities and also the actions and safeguards that unproved in particular is committed to in order to uh, responsibly developed advanced AI. They haven't developed A S L four. And this book post is sort of providing hypothetical ideas of what, uh, issues four models that get into that more advanced level might be and how you might medicate foes.

A building on the recent report, I think we discuss about a spooky evaluations for frontier models so as blackboard, uh kantele tui zis. The idea that in general these things would fall into the area of sabotage would have, uh, implications for catastrophic missed risk as well as autonomists replication, careba lies and autonomous A I research. So basically a model can go off, do stuff that you want, make itself super powerful and maybe, uh castrol hicks destroy humanity. Uh, it's quite a long peace. Uh detAiling sort of hypothetical approaches to dealing with with scenarios and released with idea to get some feedback, some conversations going uh while a sort of I guess expLoring the idea of A S L four and jar me of course you certainly have more sand this one no well.

I me so I thought this was a really interesting read. I think anthropic is um uh you know doing a good job I will say like unlike open a eye IT seems thinking in public about um what serve autonomy like intense levels of autonomy starts to look like um you one of the things that the uh i've taught quite a few people that anthropic about is how uncertainty are about what to do at A S L four.

Like once you get to systems that are genuinely autonomists. Um they've committed, by the way, to having a plan for A L war by the time they have their first model that meets the A L three threshold. so.

Which which could be imminent like a IT could happen this year um very likely to happen next year uh, according to a bunch of people I talk to and then also draw I think in is podcast with legs to share that too. Um so they need to have A S L four online soon. This is the kind of one of their first sketches attempts to do this.

And they are confused internally about what exactly A S L will require. The reason they are confused in part, is that we currently lack the techniques we need to successfully audit um the behavior of A I models at that level of capability. And this comes through in a lot of the the safety safety case sketches you to say that three dunch fast that they share here so they can have three different sketches of scenarios that they consider to be quite plausible that that you know we could run into in in coming years as we approach that. A S L four threshold, I just mention one. Um so this is our k city and mechanistic controvert ability. The thing I like about this, by the way, is IT gives you a good sense of the bag of tricks they expect to lean into once you get to the stage right once you get to the stage of models that our autonomous enough um have a long term uh understanding of their goals and an understanding that it's to their vantage to do things like power seeking, right, like hold off on displaying certain capabilities while there being evaluated so that they can be deployed and have access to a wider action space so they can ultimately achieve their goals in more in more extensive ways.

And so what they're looking at is using space auto encoder that comes across very clearly that something that i've heard a lot about, not just an anthropic elsewhere, other lads are thinking about the student anthropic been really um uh kind of ahead the game on essays basic this is we've talked about a lot but a way pRobing at the um at the structure, the activation space of your model and identifying you is the model may maybe uh activating certain concepts, religion of deception and and and cape things like that and essentially leaning on that as as one one measured that's one theme a auto interpretative techniques scale really well at something that really focused on as well um and ah and they're looking at as well activation engineering r right we were talking about that about two years ago I think when the first um cheese maze paper came out when alex turn, I think IT was posted this on an obscure blog. We we talked about that and at the time that was already clear. That was going to be part of not the diagnostic necessarily, but now they are looking to use IT for that, but also the mitigation, the actual you kind of a corrective action if you do spot indications of of deception, of of bad behavior in your models, like how how you can steer, use activation, steering to to change that behavior.

I thought that was really cool to see a pop up concretely here. They they also the case study three is also really interesting. Um they talk about kind of figuring out when your model um might be relatively benie after training and then after fine tuning, you essentially cause one of its some personas to emerge.

And so so the idea behind this is again something we've talked about. It's fairly nuances, but because uh language models are generally retrained with cross rop OS um cross android y losses is what's known as a mass covering objective. In other words, IT assigns some amount of probability to all plausible completions of a sequence rather than just focusing on the most likely one.

And um IT has that that property because IT penalises the model heavily for signing really low probabilities to sequences that do end up occurring. So you want that feature in contrast to other objectives that could encourage the model, say, to focus all its probability on just a single best completion. And that's relevant because they're arguing that the free training naturally leads to models that can exhibit a lot of different behaviors or personas because the model needs to maintain some probability on all the different ways the text could plausibly continue.

So it's gotta hedged bets. And naturally, that means it's got to be a composition of sub models, each of which could could look at alternative strategies. So there's a risk, if that's the case, that through post training, you may end up amplifying one of these um one of these kind of personas or or sub models that are more capable or or inclined towards strategic deception.

So they talk about their mitigation strategy, all that stuff. None of this is final. None of this is meant to be like a vi game plan, but I think it's in a really hint of what could be the game plan as you approach you more, more automated researcher type like drop in replacement researchers, which I think is where we're ultimately we're headed.

but kind of cool. And next, shifting more into U. S. policy. We've got a do of stories dealing with the tsc fab zona and the chip sex.

So the first story is dealing with a finalized deal of funding under which chip sex for this fab. We are told that the uh T, S M C will get six point six indirect funding for a fab and also five billion and long guarantees. And under his deal, S M C is committed to make chips of its most advanced production node at twenty to date, a sixteen in the U.

S, although IT will only happen around the end of a decade, three years after IT enter as mass production in taiwan. And a related story that also came out is that the minister economic affairs, uh guess in conversation mentioned that under current rules, cmc cannot make their most advanced chips abroad. So there is actually a law or rule that uh the whatever they are producing abroad has to be a generation behind that whatever is being produced in taiwan. So I suppose IT seems like stories go hand in hand.

Yeah it's a it's kind of funny because it's your taiwan serve domestic policy is designed to make sure that the earth matters on a nice edges. When IT comes to the Sandy conductor supply chain, they're like, no, do there's a risk taiwan is going to be invaded by china and that our fabs are going to be knocked out and that the word semi ductor supply chain gets new as a result.

Um and so naturally the united states and other countries want to onshore uh even T S M C facilities right get us to build fabs in their countries so they're more robust. Um but we don't want that. We don't want that because that takes away our leverage.

That means that the most advanced ws are not being made in taiwan. And now if they have an actual like policy uh to uh to enforce that, which uh you is kind of interesting now the way he freeze IT was we will not be producing two nanometre chips abroad. Um this is as you said, it's not actually just the two nine metre ship.

Eventually they will. In fact, there are plans to do that, right? So T S M C S first terrazas a fab there um that is going to be a foreign iy or fab um which is actually starting up pretty soon in the next couple weeks.

But there is a second fab and the third and both of those are actually going to be like reaching all the way to tune animex and beyond and and they're going to come online more than the eight era。 So if we are eventually going to get two enemies are in the united states, but we are not necessarily going to yet the leading node under chinese law, by bike, sorry, under tyrannise law by then um you're going to have the next generation ships online, eight, sixteen and so on. Um so this was made in response apparently to concerns that T S M C might be forced to produce some of these chips in arizona ahead of schedule.

Um once once donor trump was reelected and know who knows but uh IT is interesting how how the election is having an impact. They are claiming T S M C that there is no impact, that all this is kind of like business is a 说 know whatever。 But I suspect that there's going to be some, some tougher negotiating around things like .

this in the future. And a couple more stories. This one I call back to something we already covered in the subsoil, the because OpenAI two percent plans for U.

S. A. I. Strategy and allies to compete with china.

So we are mentioned how, as part of this, a policy proposal or policy discussion opening, I made a pitch for a hundred billion dollar, not a center. And as you mentioned, a germy. This is kind of an overall blueprint.

And IT does propose some things that sounds like acts like laws. So for instance, there is a national transmission highway act to enhance power fiber uh, connectivity and natural gas pipelines. And they also have this north american compact for A I that would form an economic book meant to compete with china. Uh, so these are, you know I guess, opening I is getting into the game of trying to shape policy with these kinds of ideas yeah one of the .

interesting thing. So the overall they're called the blueprint for A I infrastructure. And um there they're talking about setting up artificial intelligence economic zones and tapping the navy's nuclear power experience right to um support with essentially getting government projects for power that would also be funded by private investors.

So looking at how a lion incentives there, which is really good, like the U. S, needs to rethink its energy strategy. A big, big, big, big way you ve got like A I is set to be your double digit percentages of the U S.

Is total economic demand in the coming years, like by twenty thirty is starting from basically not wealth from more like four percent right now for dat centres at large. And and that that may that may actually accelerated even faster. The only thing that happens if you don't build energy infrastructure in the united states is these data centers will be built elsewhere.

That is what will happen. We're already seeing that happen or at least the fitted with that with the U A and another kind of uh um autographs or governments like that, you don't want to be building your key national security infrastructure there. Um I will say I mean opening eye in in kind of at this point pretty typical self serving nature um is is picking their their arguments pretty carefully here.

So they made the case that this would be great for a for job creation. We create tens of thousands of jobs um and he would boost G D P G growth ly to modernize graal that's uh I had a nice little chickle about that just because open a eye is um they're looking at automating the entire economy. So sure sure that's that's the game plan.

Yeah all of those jobs. Um anyway in may, in the very short term, create those. But the bottom line is that that's not where the future is added. Um a lot of bombastic and I think accurate stuff, they say as foundational technology, as electricity A I will be and promising similar distributed access and benefits.

Baba I mean can see them uh tweak their language in real time to be more republican coded after spending years and years um playing the the like the the democratic side of the ticket. So just using to see this happen but um anyway, it's it's a IT is there's a lot of thoughtful stuff there, but he is very obviously self serving. And and their um conditions that surely ought to be tied to building the kind of in infrastructure we're talking about here, right?

Like the security situation, these labs of visual opening in particular has shown itself to be just not interested in engaging with like whistle blower concerns over security, whether you look at a plash can berner or any the number things that even we highlighted in our report uh from from last year, from early this year. So I think you know you ve got a you ve got to care and stick IT yes, offer um your power infrastructure, but it's gotta come with concrete requirements for heightened security that these labs and again, i'll say opening particular has historically shown it's it's interested in doing IT to to the extent that IT allows them to get by congress and and to have decent relationships with the executive and so on. But um really um not on the kind of cultural fundamental level among the executives of the lab in. And I think you got a look at Samuelsen .

for that in particular and not the last story, again, dealing with open ee and going back to safety open, I loses another lead safety researcher, lilly wang. So that's preventive story. There was this announcement by a lion that last day will be november fifteen, a long time employee at open the eye seven years was a leader in the safety team. Also, I would say, famous to me as the offer of many very detailed the blog posts.

Robotics, yeah.

A little log, very deep. Great overviews of A I research. So again, we we don't know how much to read the interview. Of course, it's a part of a trend of safety figures, uh, leaving opening eye. This could mean whether some internal disagreements that could also mean that this was a very time to move on after many years, but I were away. This is of the news on this front.

That's sure. I mean, I look at her ten ten you know seven years, it's quite something. Um so yeah could just be that it's also the case though that um if if you think that your lab is on track to build a agi, whatever and and you're on the safety team, um you you probably you try to stick around almost no matter how painful that is a if you think that that you have any kind of influence, if you think that people will listen to.

And I think certainly what i've heard talking to whistleblowers that open eye, some of whom have left recently, uh the feeling is very much like you. They were disappointed then this happened on the safety side, happened on the security side. Um just that the systemic um of systemic ignoring uh these these national security concerns that Frankly are raised the and trying to trying to shut them down. So I think in that context, IT could be another instance of that trend of just like nobody's listening to me when I say that we have no the ccp being a threatened in the sand and so um i'm leave or we're not on tractive to solve our control problems, whatever, or IT could just be it's been seven years so so hard to know. And you know speculator is .

going to speculate exactly. I guess if we find out what wing moves out, your topi C2Be a s to ry tha t's rig ht and tha t doe s IT for thi s epi sode tha t we did hit a p ro bably a l it tle mor e abo ut hal f, but still not bad. I will put ourselves in the back and this one.

So thank you for listening as always. Uh, you can always go to last week in that a eye for the taxes that are also for the links to the news recovered also in the episode's ript. As always, we appreciate your comments, your reviews, your sharing of a podcast with friends and coworkers at SATA. But more than anything, we appreciate you sitting in, so keep doing that and enjoy this ultra song.

the. Circe chance.

This the free say, I say.

world.

Seven go chain and breaking to me.

Trains with the logic, the street in the.

冰淇淋 the song。

We find our place last weekend. The eyes you will be in break.

来， 在 下面。

Last week in A I A week. Should take this.

See you like your dream tonight 不在 一起。