#188 - ChatGPT+Search, OpenAI+AMD, SimpleQA, π0
01:51:50 Share

In a world protects on the rise. We've got the latest scoop open your eyes from ChatGPT searches with and welcome to last weekend.

Hear about what's going on with A I and as always in the subsoil, we will be summarizing and sing some of last week's most interesting A I news and as always, you can also check out our text news letter at last week and A I with even more A I news we will not be touching on in here. I am one of your host hundred crank. My backup is that I studied A I at stanford and I now work at a genova. I start up and everybody .

my name is Jerry Harris. My background is that I just had a baby and for that reason that's on my whole background but know it's a particularly relevant one.

Um so we were going na record this episode last week on on friday as we Normally do doing on tuesday because on very, very generously bumped IT when I found out I was going to be spending like about twenty four hours in the hospital with my new where everything's fine, everything is totally fine just to send a newborn e scare. But for that reason i'm sort of playing catch up a little bit. You're going to get my takes a little bit hotter, a little bit spicer. Less less baked as what I guess those are kind of contradiction tory things say you get that you get my metaphors .

um that's precompiling more runtime yeah .

that's right yeah I like inference time compute over training time what noodge works for you that's be doing um yeah so you know under very diligent ded a under stories like from between the hospital stay and now and so i'll be flying by and see to my pants on that stuff um I do apologized you you'll get to see I guess my my terrible thought process in action um super excited to be here and uh yeah like guess so much stuff did happen so there is there is a lot to cover that's right.

Yeah let me do a week uh, preview of the episode. So it's it's a bit of a mixed bag of stuff this week. There's no like one unifying theme who was an apps of big story, of course, a search in chagas.

T and we also have apple intelligence applications and business a some of regular themes. You've seen that with open the eye hard to wear with author as driving and meta and other striking deals with media providers, then on to researching advancements. We will actually be fighting something in robotics, but i'm excited about and also some evaluation type things.

Policy and safety are usual kind of grab bag of different things in some opinions as to policy and also some research. That's pretty interesting. So that's the preview. But before we start on reviews, real quick, as always, do wanna give a shout out to the listeners who provide a feedback and comments. I do look at you to and try to sort of be aware of anything people say on there. And I do also periodical check reviews on apple podcast, and it's been really nice to see a few more come in in fact to one of them uh actually can graduate jermy on your newborn e daughter .

which is a very sweet .

yeah thank you now forever part of the uh apple podcast review ah and it's also interesting in these reviews to hear about people's backgrounds. So there's a couple of of these that say, for instance, there's an immigration lawyer and founded a company making AI products for legal professional, uh, there is a senior tech and finance executive. Stuff like that is actually really interesting to see the kinds of people that are interested in this and are benefiting from IT yeah it's it's .

actually kind of wild um when you all run into IT in my line of working different ways for people to say, oh yeah you know heard about this ball in like yeah that is funny we discovered that on this podcast but yeah that's where I heard IT um which is always kind of amusing and cool so really do appreciate the reach out and um IT i'll tell you I I mentioned the setting last week but IT actually feels like a community um you know like the number messages that they came out like you know good luck with the baby and sort of thing is very, very sweet, thoughtful and anyway makes so much easier to to put in the like five hours or whatever is to prepare for each episode and we really do appreciate a club IT some .

people yeah I don't know. Maybe will think about starting this something if people want to make a last week in our community and discuss say I knew i'm just throning out there totta you know, a top of rain, but if you would like that, you can figure, let us snow. And then one last thing before diving into the news, as with the last episode, we do have a sponsor to acknowledge dge, and that sponsor is the generator.

And into disciplinary air lab focused on entrepreneurs AI from box and college psychology, if you don't know, is the number one school for entrepreneur ship in the U. S. And this is a new initiative with professors from different departments there organizing of this, a new kind of introduction lab that has various groups focusing on, for instance, A I and interpretation, urban business innovation, A I F ics and society, the future and work a talent.

These sorts of things are, they also focus things like peer training people across bosons faculty. They are fans of a podcast, actually, and is a cool sponsor, as I said last episode, because there's no product here for you to buy. I just wanna go to a spread awareness about this new initiative.

And this week, where is a fun thing, we didn't want to shut out. Some students awareness are leading this event. A I build a fan with microsoft where rail be doing stuff on interpret ship N A I.

So these source of good initiatives, guess will be calling out and um just making sure people know what this is a thing. Oh right. Well, that's IT.

Let's dive in to the news, starting as usual, tools and apps. And the first story, as I previewed will be search in ChatGPT. So opening I has had search GPT in beta as a sort of a preview, and they have now launched live as a feature in charge beauty. So the way with the work is, uh chage beauty can itself decide to search for information across the web to answer a quality, or you can manually tell to search for stuff on the web when IT replies to your chat. So it's actually directly built into a Normal chat.

The experience there is no separate thing like search GPT to use, and it's very much similar to things like complexity, to things like A I overviews from google, these other kind of offerings that let you input a quality vulgus will find the relevant news stories from various sources on web, and then that will be fed into the alarm to be able to generate response with awareness of these various new sources. And as with these other things, of course, it'll be citing its sources, letting you know of the links to go back to the original and read about them. So as whenever things this is pretty important uh, tragedy tey knowledge was limited to the past to like twenty twenty three, I believe and so now I will be able to, uh, talk to you about kind of anything what's going on now I suppose yeah there .

a couple of interesting little tidbits too dropped into the so one is when I asked what the technology behind this, the models behind this were there is A A rap who said, well, it's a mix of technologies is including microsoft being so that sort of an interesting um point of deeper integration between open eye and I mean like so so microsoft in right is really open a eyes some version of GPT four that's probably been tweet. But one of the interesting things here is like it's unclear when this crosses the blood brain barrier like is open. I actually using um like some some propriety microsoft uh product.

And if so, that kind of deepens the dependency of opening eye on microsoft, right? If search GPT becomes a breakaway product, if IT allows open eye to take a by sweet wet you like multitrillion dollar uh search revenue market, then that's A A really interesting thing to to bind itself to open a to microsoft with um so there is that there is also the fact apparently the underlying search model is a fine tune of GPT four o so at least one part of the stack here includes that um and did an initial test or like ten thousand users and and seems to have gone well. Another doing the wider release.

Interesting that they're choosing the stack GPT four o so we're going to proudly ly get that multimodal functionality as well, a kind of more and more deeply integrated over time. Um but right now that that seems to be where they're going using existing technologies. And you maybe a partnership with microsoft to access being to kind of supplement all this because microsoft has all that experience um questions, of course, as usual, right?

Any time we get into generative search, what about the Price point? What is the cost going to be to open the eyes to serve these models? Because gender of search is so much more expensive if you're actually generating tax and that's that's going to be figured out.

They say currently there's no plan to advertise in ChatGPT this i'll be through the ChatGPT in your face so obviously like no it's not like you want to search dot ChatGPT dom or whatever you're just doing ChatGPT and then within that, your searches will be ready appropriately. Um but they are flagged. There are going to be some limits on how often people can use the free version of the search tool.

So again, there is a bit of a question is still like whether this is google or whether it's perplexity right. Are you going to are you going to pay for a certain level of search? Like how does that all shake out and then how our ads going to be displayed in all that stuff? Um anyway, kind of interesting as well as they point out in the article that this is chopping uh, just a couple days before today, which is the presidential election, the united states um you know the need for that is interesting marketing choice really because if things go wrong with general search, say today and there is a big story like that, sets of pretty risky time to be doing this when presume ly, they could have just waited a weekend and lunch at them but you know I guess opening eye is choose in the ship as they so often do um and and let me .

have IT right and you know maybe we decided to just do IT last week. So if it's not a clipsed by other news in the U S. Be part of that, that doesn't sound right.

I mean, opening, I would never make these kinds of decisions on the basis of .

marketing. Yeah I look they are now business that cares about you are. Um and actually on that note, it's funny because I did try IT out a little bit I compared to a plexus and tragedy of search on informing myself about the local candidates and elections so that I will need to be voting for in polo auto.

So I just know there is a lot of like candidates for city council. There's like twelve of them. It's going hard to get over by. So it's actually both of these things I found quite useful for that. I would say perplexity is right now Better for these sorts of a more intricate research task to require sumi zing, a lot of information, tragedy ity decent, but didn't present to information quite as well. At the same time, if you are already using tragedy ty and paying for IT, I could definitely see if this being A A reason for you to not pay for perplexity, which could be a real problem for a perplexity.

I guess will see you searching up.

Next up, apple intelligence, where features are now rolling out in beta. So this is a developer beta version of our S H eighteen point two. And there are these up on intelligence features that are set up to be publicly available next week, I guess probably this week.

As we have seen in of the features have things like integrated writing tools, image clean up, article summarize and a redesign city experience with typing input. Oh, this is not the smarter theory of L M, uh. And then the developed beta users can access extra features like gene mogi, image playground, visual intelligence, image round and ChatGPT integration that were to users a bit later.

So we avoided covered this, I suppose that to announcement time that apple's approach to A I is to not have a sort of one big thing in their I O S. acquisition. Instead, having a yi can across there was in various ways kind of as features is really more than one big A I. And now you're seeing some of features are come out.

Yeah, it's also that they seem be going for much more that platform player, right? Like it's it's very apple to do this, be the integrator, a lean on the hardware as the thing that's driving value and and there's a bit of the and that's A A microsoft question.

I guess you know there is this this notion that you want to commoditize the compliment to your technology, right? So famously, microsoft um they make your Operating systems and software for pcs. The compliment to that, which is a big counter, intuit is the hardware that actually runs IT.

So you make dirt sheep hardware and what where you really get your margins is on the software. Um and apple, you know in a certain sense here, maybe maybe going for the opposite play as they had been with higher hardware. I kind of make them a more natural integration point for a wide range of different technologies is not just leading with, hey, it's just apple products. So that's why they have a lot of partnerships with the likes of open eye. And anyway, so interesting kind of further pushing that direction strategically .

for apple onto the lightning ground. And speaking of that relationship between open the eyes and microsoft, the story we have is that github copilot will not support models from anthropic, google and open a eye. So as a developer, get up a copilot is for coding.

IT gives you suggestions, A I suggestions as you write code. Now as a developer, you can choose whatever model you want from things like cloud three point five games at one point five pro and the opening eyes gp for all, or one preview and one mini h. This is interesting because get up a copy so far has had just one.

The model there is no ability to choose. And that model seemingly was built on top of opening technology, trained with data from across github. So this kind of allows microsoft to be pals with more companies, I guess.

Yeah, I think this is more evidence for I am biased and beating this drum for the idea that your L M. Class is increasingly becoming commoditize. great.

So you're starting to see the leverage shift from the frontier companies to the aggregators. So you know, like you've got open an eye anthropic, you've got a google products, all these competing on one platform. And you can almost feel in that moment it's becoming cheaper and cheaper to just switch between them.

And what does that? Well, IT drives your margins if you are those companies down to zero, right? That competition just roads your margins and then puts more pressure on you to release the next generation of product to desperate try to close your way ahead. The competition arguably, you know, that's been one of the big things driven even anthropic now to release more, more powerful systems in spite of uh earlier commitments, uh mi commitments, ambiguous commitment. Ts certainly interpreted that way to serve lag or or come close to lagging uh the true frontier of the space.

So ah you know this is a big amount pressure to put on, even open the eyes as well as they roll out their own one mini which is now available in the sweet of products a long sign cloth three point five senate which just came out and gami one point five pro from google. So yeah this is all kind of increasingly it's great for the end and user, at least for now when IT comes to competitive dynamics around this stuff, again, margins get crushed. And if you are a small company trying to build your your bespoke aleem, whatever, I think this is a bit of a warning.

I've been beating the trump for a while, but I think companies like coca really gona struggle. I I don't think there room for meza scope c players. They can benefit from economies of scale and and just having massive cloud provider supporting them.

And they were not just seen that this is play out. One other note from this too, so so anthropic had a whole post about the inclusion of senate called three point five senate in this whole sweet. We've covered that before.

You really, really great model. Uh, IT is soda on a software sweet bench verified basically this opening eye tweet version of the sweet bench benchmark um which is biggest coating soft engineering benchmark um so anthropic sorry, club three point five minute is amazing there. The big question for anthropic has not been has never been um can they build competitive model at least for last year? So have been very cleared if they can.

The question for them is distribution. You can build the best model, the world. You can build the best product in the world. But if you just get out distributed by competitor, that can just crush you as well. Everybody sort of open a eye.

Very few people have heard of anthropic at least if you look at the average person actually the average person play has heard opening I much too but you know what I mean um and so this is sort of like what happened with you know slack and microsoft teams. You slack had crazy growth. You're doing amazing.

And then microsoft teams launched to initially, initially a fairly mediocre product. But it's microsoft. They are in all the Operating system. So they just had massive distribution. In a sense.

This is anthropic piggy backing off of github s amazing distribution and this kind of levels of the playing field a little bit between them in OpenAIr, at least when IT comes to software development. So that's a really interesting and development. It's also interesting because github, of course, is owned by microsoft. Um so this is Normally you would expect that to be a serve opening, I friendly partner, but here they are platforming anthropic and google alongside open eye and creating this sort of very direct had on competition and giving anthropic distribution that they they really, really need and that that does level the playing field. Ld in a very subbed al way.

And speaking of coding, the next story is related to that and it's about unprofited as well. So lad, the child bott from one topic has now a new feature and analyst tool that will allow us to write and run a script code. And this is important because now if you, for instance, upload A C sv, a file with data in a uh cloud, can write some code, run IT and then output the results of that logic processing.

So that's something that's been around in charge partia belief for a while and is now being added to cloud, a very useful feature because IT makes up for weakness ness of a alarms. Alms are not able to let say, data crunch to run algo rooms. But when you let them write code, which are really very good at kind of a weakness of alarm and sometimes goes away.

yeah, this is, you can think of IT is a way of grounding the model, right? You at every junction in the reasoning process of an LLM, you have some probability of failure. And when you have a reasoning junction that forces the model to write and then execute code, the execution of that code is not subject to fluctuation.

So it's a an injection of ground truth into the process that can help you get more reliable outputs. And they are a flag this they don't explicitly tell us quite how this is working in the back end, but they say, instead of rely on abstract analysis alarm, you can systematically processor data, cleaning, expLoring, analyzed, step by step until IT reaches the correct result. This starts to sound a lot like, well, no inference time computer.

Obviously, that's what's going on here in some form. We don't know exactly what IT is. What kind of prompting a strategy is just an agent's cafod? Is IT a model? You know that that what's what's the training routine in the background?

We don't know, but there's obviously you know more and more leaning in inference, time compute and in this case, with a view to solving very specific data analytics problem. So interesting that they box that out as A A categories of a feature that they want launch that makes all the sense in the world because of the have a wide applicability of data analysis. But yeah so we'll see if he gets uptake. They give a bunch of use cases as well in the post that you can check .

out a few curious yeah a speaking of distribution and get awareness. They do like explicit call out a marketers, sales team, product managers, this is useful for you ah and they have this little video of that shows not just doing some processing, actually showcasing a whole chart based on some data. So certain ly useful if you work with data.

Now moving away from chatbot, we have a story related to eleven labs which does a generation of I O of uh A I speech and they have any feature that is a voice design that allows you to generate a unique voice from a text prompt alone. So eleven ABS is the leading a company providing a text to speech functionality. You can some speech and get audio that is very, very realistically sounding.

So far, that was limited to a set of voices they provide or you could. Train your own by giving you a bunch of data. Now you can create a voice just from describing, so you can say you like serious new aster that is a real professional or something, and that would generate a voice. So another way to make IT easier to make this do what you want.

Yeah love to love to see that training data. Sep, but yeah, it's also is one of these things where we look at multimodal products like this. I often wonder how the the future of prompting is gonna, right? Because explaining verbally what kind of voice you want to get out of a system like you know text data is it's not designed to IT.

It's not very readily able to kind of give you get across what you mean by a voice, right? So we do we do impressions really to do that, right? Like how would you describe b commoner Harris voice, Donald trumps voice? It's really difficult in words.

And and I so I kind of wonder if if there's a uh, more of an iterative cycle that you'd want to get into with these things down the line where you know generate initial one and then be able to take that as a template and give a subsequent kind of feedback or we're doing impression or something like that get give the input to be multimodal. I imagine just like with images, right, where you often want to upload A A starting point image um and then start to modify IT like I could see that working with eleven lives products. But but anyway is an interesting new form factor for for prompting for a generate .

voice models yeah and that's a good point. I think the idea is more so this guy who is speaking how they speak and then ah we how they speak is implied. They have an example of like evil over or something.

Uh, I will mention I did try to see if this can be abused. I copied the description of rack obama, like the first paragraph from ukip, dif of out, including name and eleven lives, did cash me and didn't produce a voice of obama. Guess good.

Now I want you to follow last week. And I A pocket, so got a lot of good folks. And on a beautiful smile and beyond.

we have mid journey's new web editor. Less you tweak images uploaded from your PC. So my journey so far has been a pure image generator.

There has been other competitors are allowed to the images and edit them. Uh, and now you can do that majority as well. So on this editor, you can edit that would allow for resizing racing and things like that. And the texture, which lets you modify image contents with prompts, this is still in beds limited to about a its limited to users who have generated at least ten thousand images and have annual memberships and have been mostly subscribers for a past two of months. So I guess there was some very, very loyal majority customers where yeah .

and I actually like that. This the first time i've ever seen, uh, close by a kind of d market and quite that way, it's almost like a loyalty test like hey have you know actually been using IT like an obsessive just I think it's brilliant. I mean backroom y combination.

This is one of things they tell you to do right find you're like um it's more important to make a hundred people love love love your product than ten thousand people like a decent and when you find those people, you figure out why they love your product and you double click on IT. So I think that may be part of the thinking here but so so interesting um i'm sure there have been examples of this that have happened before. I haven't see ni yet um but yeah and then they share this kind interesting screen you know famous picture of the a the uh zebra crossing with the beetles of walking across IT. Well, they they play with that and basically give you a new background and pretty, pretty effective. So just goes to show, I guess theyve got they've got a pretty interesting new feature here and an interesting .

new with a launch IT. And for the last story, actually speaking of my journey, the headline of this article is watch out. Main journey we craft just announced a new AI image generator model.

So yes, we craft is a company that focuses on things are design, and they have this model. We craft the free, which is pretty good. IT came out for a while. IT had a high ranking on hugging faces text to image model leaderboard and was actually a kind of secretive people who are wondering what IT was. Now we know IT is a small to request to you free, which is impressively good um a generation of of course, if you take a look, you know it's kind of hard to distinguish, at least for me at to this point between the different image generators.

Yeah that's that's really it's been a problem kind of the the quantitative tive side of image generation for a while, right? How do you actually measure the quality of generated image? It's text went through that phase as well. We're like blue scores or just like not enough anymore to to get us past that point. Once your model gets Better than a certain kind of human, a certain quantity of threshold, anyway, IT starts to be really difficult measure.

So i've personally found this like I am still, and this is a function, I guess, the fact that i'm not a graphic designer, I don't actually use these models to do specific graphic design stuff or or animation stuff. And i'm sure if if I were in that space, i'd be like, oh yeah, you know there's one model to lot Better than the others. But I think what what just looks to the the the untrained eye, like a dead heat across the board, I think we're going to start to see that more and more in the domain specific applications.

You know, six months a year from now, i'm always prettily interested in and and confused by the massive of VC investments in this space when IT seems to be rushing towards commoditization so fast. But who knows? We will see if, as Peter deal would say, if competition is for losers, or or if actually that I .

yeah exactly. And the to that point to be crafts found their emphasizes and models design centric approach, which aims to give designers control over their output. So I think the differences there will be like that last one percent at last five percent. The quality as important to people who are you use of this professionally and that kind of interesting, we are beyond the point of just quality you now need to really get into in the integration.

Actually, I think such a great flag to for the the place we are in eye across the board. I think in industry, and I think a lot of people, and this is the source of a lot of confusion over is the space over hyped and not right. We have these impressive demos. You get to say the ChatGPT moment people realize all there's all this low hang fruit and and there is but often for the most valuable applications.

It's that last mile, right? That like last one percent, it's no good to make a you know like a an APP building model that makes what a mistake one percent of the time or staring, maybe not that bad example, but anyway, a model serves critical user experience because one percent can be way, way too high. You know, you'd be surprised a AA lot of the time that's an issue.

And for agents, models that have to string together a coherent set of actions, one percent array, like will nuclei, if your thing involves you know twenty steps and whatever the number is like three person of the time, that means you'll be screwing up along the way. Um and so yeah actually really be less than doesn't matter. But the point is, you know, I think what's going to be surprising is you ve got this initial thing where people think it's over hyped because I can't solve the thing.

But then suddenly you just get to the point where as you scale, uh, essentially you're driving down that area rate and the new cross magic threshhold, all the sudden things just become possible. So I think it's possible for these things to be both in a way over height and under height at the same time. Just a question of the the time line you look at IT on. And I think and that's the the big question with these these image models and more and more thic agented systems.

we're going to see that moving on to applications and begin with a trend we've been covering for a while this year, big company ies striking deals with media publishers. And this time it's meta striking a multi year deal with reid's, in case you don't know, is a very big distributor of news.

A lot of kind of breaking news goes out on roaders, and this would allow mas chatbot to have access to real time news and information from reuters when the users ask questions about news or current events. Unclear if this also provides access to training formatter. But the regardless certainly seems definition of the partnership that open I has made with many, many uh, media companies like royal yeah it's it's all part .

of a broadstone to when when he comes to meta of trying to avoid or historical, they're tried to avoid having hard news on their platforms. Your taking some steps to avoid the current event stuff more more leaving that maybe to platforms like x um in fact, in this case, we've had their executives to come out and said that they're not wanted quotes to do anything to encourage hard news and plenty of content um but but that, of course, is part of what's going to be nudged towards here when you start to explicit add news into the platform.

Um and anyway, so there are all kinds of questions flowing around around what are they going to do for for content moderation for for generation of these these responses that deal with news and current events are no no comments on that yet um kind of understandable given that Frankly provides an unsolvable problem given today's technology. Um but we also don't know what the terms of the deal are. So we know there's onna be some kind of compensation to writers, but whether this is an annual licensing model or a revenue share thing and how much money we just don't know.

Um so kind of interesting to see that that next step um they also flag there's a uh interesting note at the bottom these articles that talks about um how meta though meta now appears to be willing to pay for news content is also simultaneously fighting laws that would require a compensating news publishers for their content on social media is this interesting kind of economy right? They want to find ways to license with individual outlets, but they don't want to just blank IT, you know do revenue sharing for clicks and things like that. There's been a very high profile situation up in canada actually where you literally cannot access news on facebook and instagram.

Because the governments tried to get meter to pony up money. Basically this bit can then restarting late around traditional media. That's a very controversial proposal as well.

I mean, if you ask me, it's it's not the not the best idea, but it's LED to matter, just like outright withdrawing from the canadian ecosystem altogether. So kind of interesting similar things happening california too. So this seems like the opening eyes play right. People are just saying, hey, that looks like we can get a safe harbor by a licensing with um with these news outlets at the very least kind of installation ourselves from charges that we're just rapidly like scraping news with with no regard for people's copyright um copyrights um so yeah a bit of A C Y A manual but yeah we will see if I is sticks and if other companies follow .

living right along. Another story really to a trend that we've been talking about all year. It's OpenAI and their search for hardware.

And the story is OpenAI will start using A M D chips and could make its own AI hardware in twenty twenty six. So a opening eyes, collaborating of outcome on these custom silicon a chips. And they are integrating A M D chips into their microsoft asia infrastructure.

And these are things like M D M I three hundred chips. Uh, I suppose previously, of course, in video was a big company that everyone went to for a AI compute. But now AMD is starting to be a real competition in the space.

Similar yeah the reason behind this is partly at least opening eye wants just more more flow. They want more um sources of of that hardware. You know they're not going to say no to more NVIDIA hardware, but if there is another source, H A A M D that's offering some great will take IT also diversifying your suppliers beyond uh in video gives you more leverage in the surprises gotham side or or the ah the anyway the acquisition side.

Um it's it's also part of opening eye, just a shamelessly recruited poaching the top part where engineers they can get at google. So theyve assemble the apparently chip teams about twenty people. And these these are the people who focused on building the TPU at google, the flagship specialized asic that google a builds the tensor processing unit um and ah and so the they're obviously interested in fairly custom hardware, which is its own really interesting story.

As you see more and more the the research of the frontier labs like opening, I like an anthropic like google start to go to dark, right? It's no longer open sore stuff. Everybody's hiding their trade secrets um from everybody except the chinese that is um they they're starting to build these out and you're starting to see more, more differences between the approaches taken by these labs.

And I think that's really interesting. We need to start to see the correlation in the in the kind of ei model architectures. And one consequent of that is you're going start to see the correlation at the hardware level and open a eye kind of characteristically so aggressive in terms of pushing forward on on all fronts, including hardware, basically making a big bet that like hey, let's slow double down on building our own custom hardware.

They have to partner with broad comes they don't have all the capacity um to design in house h. And so the first time we ve heard of about that opening, I brought from partnership, by the way, that's something we covered previously. But this does move broadcom further into the camp of competing with video h. We're on the design of these kind of advanced assets, which is something that broadcom has done historically. But now these are really A I asic meant to sort compete with this a TPU model or know the G P U models that that in video pumps out.

So that's a really interesting um and ah we'll see it's it's also in some ways um not the most surprising thing uh because we've seen a whole bunch of news about open a eye poaching hardware engineers uh from google in particular and fascinating ly broadcom actually had previously been google TPU partner and something that hasn't really been caught on to a lot of these reports. But IT is the case that initially, when google was first designing the TPU, they did tap Brown for that effort. So this is actually opening.

I mirrors google initial strategy. Some people have said all well, you know department with broadcom, they don't have the capacity like you. So so says something negative and I but this is how you get that for off the ground.

So IT would not at all be surprising if you would if you would see that happening. Especially you've got actually is a great comment on hacker news about this. You've got um all these these employees form a google employees moving over to open eye and basically bringing that vender relationship with broadcom with them as well. So there's a strong bias now to just replicate what worked so well. Google with T P S over on the open I side.

right? exactly. So the partner ship broking is very directly implying that they want to build something like with TPU with thanks for processing unit that google said since I believe twenty sixteen was version while and I have iterated on a lot since then, broken SHE has generated or has gone billions in revenue from that partnership of google.

So makes sense. We want to do more of that. And according to vous, news opening has been working for months of broadcasting to build their first AI chief, focusing on the inference. And another dimension there is having a custom hardware for inference lets you be faster unless you be cheaper. And they have often talked about margins.

And we know that opening I has billions in revenue, but is still unclear whether that can convert to profit, whether you can actually be profitable while still competing in this Price war that is ongoing. We have on tropic and other providers. So I do think that this would be one of ways of that these players can stay ahead of the pack is by having hardware that lets some have good margins, which is otherwise very difficult to just gp use.

Well and specifically with the inference time compute, right? Because that's that the entire paradise, if you look at the direction that feels heading in that that's becoming dominant or seeing the shift from training time compute to inference time piute. And like you know, not not to do our own horn, but it's been about two years that we've been talking about.

This shift was coming as soon as the earliest, earliest hints of scaling laws for infront time compute and insert, like exchange rates, tween training time computer and inference time computer, you could trade off one for the other uncertain context. Um and I think that's that's where things are going so opening eyes saying, hey, you know what with models like a one which are explicit, our big age I play, uh we're going to find themselves using more and more influenced compute. These things are going to spend more, more time like pondering, thinking over problems as they go.

And why optimize hardware to be able to do back propagation? You know there was to be able to do like model training when you're only going to use a relative ly small fraction of your hardware dollars to to do training. You know if if you have a more specialized problem like inference, you can make a more specialized piece of A I hardware and and have, as you say, Better margins. And obviously companies like like rock of um have been doing big plays in this direction where where it's like, okay, you we're just going to carve out the inference piece and just crush that and and it's at least seem to pay off all for them so far. So it'll be cool to see a opening eye double down on influence even more.

right? And inferences also is the main finger people pay for when you, their P, I set a right. So that's the pricing on the inference is the differentiated for large.

Like right now, there is some difference about jumpity for all song at three, five, you can go back in four the o um pretty easily. And IT IT comes down in a lot of cases, surprise. So it's very important to be able to, I guess, be able to keep up with Price reductions that keep happening.

And one quick last note, I found interesting. Uh, this is a big deal. A broken ms. Stock jumped four point five percent on the news, and ams went up by three point seven. So fatalities, something about, I guess, how investors are using this on to to lighting round.

And we begin with ve, a competitor to open a ei X A I and venues, is that they are looking for funding. They want to be, uh, raising some more billions. They have raised six billion so far and you should be funding reaching evaluation of twenty four billion now.

They are seeking more money for evaluation of forty million. And IT seems to be the case of that way. He wants to raise this money to be able to uh buy more G P S, to increase their data center from one hundred thousand to two hundred thousand gp s.

That what i've read about this. So I guess unsurprising in a way that xi needs a lot of money to a try and catch up. Really we've open the eye and tropic. They seem to be essentially targeting the same customers with the same type of product. So um yeah X A I certainly making a big effort on that truth.

Yeah there's so much um is such an interesting play because you might have written off um X C I before all this right where there was a time when I was like google IT was microsoft sash opening IT was and thrown back and they seem to be the only one maybe with the the resources to pull this off, X A I has seemingly come out of nowhere.

And when we were warned about this, when we were doing our investigation last year, just before X A I had launched, they're like, hey, watch, watch these guys because the the acquisitions they're making on the the GPU from the hardware front are pretty monstrous. The the differentiator right in this space, if scaling is true, which basically, although frontier labs seem to think is the case that even scale your A G I something like that, then you have got to a number one. There is there is no prize for number two.

And so the fund raising peaches have got to sound like we're going to do IT first, that there is no other option, right? There is no fundraised sounds like we're gonna you know self improving age and we'll do its second by then it's irrelevant. Someone else already done IT you got you run what effects all that if if all the sphere is out um in that context in one has done a spectacular job of taking X A I from horse nobodies heard about to a very competitive force for the head of race, not only hitting the as he as he advertised that you know the largest supercomputer, three, eight, one hundred cluster uh train cluster that exists um which is probably was probably true at the time.

But also you're doubling the size now in an presidentially short period time will launching this cluster in nineteen days from the first eight one hundred gp u rack rolling onto the floor in the gage factory that isn't saying like that. The there's a really great interview with uh jenson wang um who's the co in video course who was talking about just how long the stuff usually takes, usually a matter of month or years one hundred and twenty two days in the case of X A I H to Operation ize this this whole system so really wild elan intimately involved in, as he does so well in the design of the hardware stack and an implementation of the build know this is really quite remarkable. They've also made some weird choices around how they wire all this up um how they how they power IT as well um but yeah, they don't use, for example, this very well established basically interconnect called in funny band that in buddy a makes.

Instead, they're using basically an ether net fabric that pretty recently came out and that's very competitive. But it's all this sort of like you can just see the fingerprints of e one getting all all in on this and and having a very collective, very unusual design to the to the cluster. So yeah, super interesting. And and that's going to be the play for fundraising we can do at first because .

we're doing IT differently. They certainly are running a lot of hardware. So it's very impressive that they're been able to basically catch up.

Their moves are not quite as good. But yeah, essentially you you can use them for a lot of stuff that would use chage B T or claud for. And certainly your mask, if nothing else, is good at running hardware companies .

yeah and acquiring, right, like one of the the big things that he does, maybe maybe more than anything design wise on on the cluster is just he's able to like like muscle around his big famous guy a miss and go to like jensen huang gue in video and go like, hey, dude, I know you've got a bunch of people like just buying you for GPU, but i'm cool and famous.

I'm going to ask you for them and i'm going to brag about how productive our partnership has been. And you've seen that really be an interesting boom for NVIDIA too, right? Like basically now you've got the evan musk are arguing the world's most successful entrepreneur coming out and say, hey, you know like this is the hardware stack that we use.

Their amazing working within videos been great and then the videos returned to favor. So know there's like non trivial marketing benefit to to both parties in in doing this, but that nonetheless is value that he wants bring to the table and spending the stuff up fast. So I think he knows exactly the value is bring to the table and and he he's using IT well, right? And this is this is what .

he's supposed to do yeah also inform ously a borrowed some GPU from tesla or redirected from GPU. So it's always it's interesting to see like the elon mosque mega CoOperation and how working .

it's elon corp.

yes. Next, a start up of that is raising the money is physical, intelligent, a robot, a specialist? And they are raising a millions from very sources, including jeff bazas.

So they have raised a four hundred million in a new financing round that's falling up from their initial seventy million seed funding just earlier this year, not too long ago. Now they are being valued approximate two billion dollars. And this is coinciding with some news of progress and made that we cover.

And just a little while so physical intelligence, you know no product yet. They are promising to build a sort of a general purpose robot brain that allows robots be very capable. And certainly, IT seems investors are very bullish on that promise. Yeah.

and they're being careful as well. You can see the hedging on the this serve hype of the initial product, understandably was saying, look, this is more of us like GPT one, which if you're old enough in the space to remember there before, there is GPT four, there is GPT three in level anyway. So GPT one was.

I think .

GPT two .

was eight. I .

was .

twenty eighty.

right? Yeah, yeah, yeah. So, okay. Well, anyway, IT was one of the one of the late teens in the twenties. And you know that nobody can really remember IT, because I was prepare demise.

But there are saying, hey, this is like a GPT don't think about IT as a ChatGPT, you know it's it's like a approve of concept. So um but they are saying, you know ChatGPT style breakthrough could come far sooner than we expect or IT could definitely be far out. So you know a lot of uncertainty.

Yeah, that's what you kind of expected in the space too, right? ChatGPT moment is a thing in A I precisely because we don't know at what level of the I capability. All the sudden you crack that critical threshold where some critical massive tasks become possible to make the thing viable from a commercial standpoint. So you just keep improving, keep improving and hope that you kind of flip that switch and is pretty binary, at least that's that's how products do tend to be. It's just a Normally it's humans iterating on IT rather than just be a pouring computer to a thing .

until something cracks. And as I said, more of that soon. A couple of more stories this time about way mo and we amo has also raise ed some money, in particular five point six billion from several investors, notably, of course, alphabet way most parent company, also google parent company.

So probably a lot of this money is kind of just coming from google to way mo a, essentially from the money printer that is google ads to this new initiative. But also we are participation from overseen a horror ds fidelity and a silver lake. And this is coming at a time when they are trying to expand to new cities like Austin and athena.

A and a lay and combo story also on way. O I figured we'd cover both. So the other story this week was that we meo has been serving over a hundred and fifty paid robot taxi rides every week, uh, so far lately. So that is up fifty percent in just two months ah. And that is notable pretty much just because the important thing to see is how can wait more scale, can IT actually expand its Operations rapidly to be able to come be competitive and you know basically reach for market before that? What does ah and yeah these are good signs on that front.

Yeah in no surprise, they're looking to as the next step increase to the graphic covered rates. That's that's how you do IT. I imagine also like the the the chAllenge of entering a new city is going to be interesting for them because you want to get people used to seeing other people in way. Mos to these driver list part is a little bit it's a little weird um so yeah anyway be I be interested to see the playbook for a launch a new city all these companies will have that um but uh yeah we mos will be a special .

interesting yeah will say now is in a ef if here crowd of the area you're very likely to see way more and like a two minute period very kind of all over the place.

Now do that like I remember um having met that feeling of wow when you're in sanford sco, when you're mountain view that kind of area, you do you see the future? I like I was I say like twenty seventeen and twenty eighteen. I remember walking around and just seeing people with airports and I knew this sounds stupid, right? Is now every everybody wears airports but back then I was not the case.

Everybody had these dangling little wire things. And and I remember thinking, like, man, this is either the weirdest place in the world, or this is what every place in the world is going to look like very soon. And it's generally that happened with bird scooters as well. Right now we all remember that. So anyway, yeah, I think it's probably a carbon jer .

was to come prety sure of the billboards with ads for A I companies will .

not become that's .

on to next section projects and open source. We have have just a couple fund stories here, starting with meta AI, which has quietly released and you think called a notebook alama, which is an open version of google notebook L M A nice little. You contrast notebook allama notebook alam, as you've covered, notebook alam is a pretty popular thing from google. It's allows you to upload files like B, D, S and so on. And then as if you have a conversation about those files, get summaries, things like that nobly recently that has become popular because you can also generate a sort of podcast episode we're having audio discussion of whatever .

is in most document. Do what what we do.

Certainly we will do a Better job course .

even if you about cking just don't even like like it's not to be disappointing, just will tell you about IT. Don't check IT out. I can't possibly is charming and .

humorous as us. I think if we feed IT the amount of notes we have, I can change right two hours. I don't think this would do to show, uh, anyways, novocure ma is similar in a way where if you provide some text, some documents to IT, you can chat about those documents, get summaries.

things like that direction where eventually you you get these really reliable uh audio generation tools like the original no book alone like I I know I think that that is quite game changing. Um I I actually use google, the noble calm, the original and dem like IT it's pretty good like yeah that ten minute podcast it's it's the this the right level of abstraction and complexity for conveying, you know more or less what you want.

I D like to see more stability of kind of control the the level of the discussion and and that's a thing. And I think that's exactly what open sourcing a tool like this would allow you to do down the line once they get to really good, you know, actual podcast generation so that I I think this is one of those products where open sourcing really will get you a lot of cool ideas implemented pretty fast because the way that you set up a podcast matters so much. And um uh you know anyway you could imagine people wanting to include a wide variety of different prompts and met instructions.

So uh I took a listen just now and um it's not quite as good as no book. A an impressive thing that is a very, very realistic sounding. There's like a little bit of a response is a minimal uh, envies generations because it's not using propria models is not what is good, but it's still pretty good.

Yeah the vide they went for with the original launch of no book alam. I can make me think if you ever heard of like the reply all podcast. Those sorts of things we have to to host and that there's this is a little bit it's a little bit battery for my, for my taste.

There's a lot of banter, and I can get a little bit like two windsor at times. But I gotto say IT definitely isn't. It's not on canny valley. Like that was the thing that amazed me. I expected a bit of a jAnnie e experience, you know, kind of like the whatever, that duke box thing, that opening eye first. But you know, the first versions said this one was like straightfor the jugular IT was IT was amazing and yeah so not surprising the open source version of this isn't quite up to enough um but uh .

soon have the open source community will take IT and run while to IT. Next story also about meta and this one is about a release of lama free point to quantized models. As we've covered, I believe many times on the podcast.

Quantize models are things where you take the um like real four model that usually has float weights. So things like one point to two, two, four, five, six eta, and you reduce the size of each of those numbers, you quantize IT. So that uses eight bits, four bits, something I got.

The kind of resolution of the weights of the model goes down, which makes IT smaller in size and makes IT use less memory. And it's important for things like phones. So here they are saying that they have these quante models that have a fifty six reduction in all size, fifty six percent and forty one percent degrees in memory usage. And these are one billion and three billion a size models.

Yeah I I think so one thing to to recognize about these quantized models, right, is that you take a big thing in quality when you so imagine you train usually like A B F sixteen, like sixteen, sixteen bits of of precision in your number and then you're going to reduce that to say four bits or or eight bits, right? You're going to take that model compression by throwing out a bunch of that information. Now your model was not trained.

IT was trained with the sixteen bit representation. IT was trained with the expectation that I could use that level of resolution. If I had been trained with the expectation that IT would be using, forced to use, forbids at inference, then IT probably would have taken a slightly different strategy, right? You can serve.

Imagine native if you had to send yourself instructions, and you know, you were given like just two sentences versus two pages, you'd truly use a different approach, right? So same idea here. Um so when you impose that limitation, you quantize post talk after the is that you you would lose a ton of of um of performance compared to doing something called quantization aware training.

So if you train with the kind of incorporate the quantization into the training process, many, many ways to do that. That's what they do here. Uh, we covered red to a story a little while back. Um he was called self compressing neural network and that was, I think, back in August, if you want to check that out the August nine episode but self compressing role networks as an example of this.

We're basically what you do is you you make one of the model parameters or so one category model providers um not just the weights, not just the biases, but the level of resolution for each weight or each bias since you actually have the model train on how accurately represents its own weights on the go and in particular, you can even imagine along the model to like completely zero out a waight right if if IT finds IT hey, you know what? I let me look at this way. Oh, you know what? I could probably get away within a year of really new a for a bit, a little bit, oh, actually one bit.

Oh, shoot, I can get rid of the the way completely, right? So you can gradually hole in on removing weights completely. With this approach, it's very powerful.

And there a whole bunch of other the straight through estimator stood like a famous one that the osha venda proposed a long time ago um that lets you uh easily like gradient updates we'll be profited through to wait through a rounding Operation, which if you're mathematically in client, you might recognize that is like, yes, that's that's part of what I have to do with this sort of like this quantization process. But anyway, bottom line is lots of ways to do this. Incorporate knowledge of the quantization into the training loop that gives you a Better result.

Combine that with Laura, which we have talked about before here, and gives you bit a final control on kind of additional capacity or a capability in the model. So I think this is really cool. I'm not aware i'm not aware of another quantization aware training, uh, process that's LED to a sort of frontier model like this.

So this is actually a really interesting in new development. I may be wrong, but I don't know. We cover a lot of news.

I don't think i've see this before. I agree. Yeah, I think it's pretty.

My kind of reception is usually you do more like display tion training. You do some more training once you quantize. So this is pretty different and quite cool.

And one more story for a section of this time about a benchmark opening. I has released a simple Q A benchmark for measuring refections ality of language models. So one of the major issues with all lamps is allus instance, where l emps.

Savings that are straight up, false or just made up. But you know sit in a way that seems like it's true. And this benchmark IT includes four thousand three hundred twenty six questions across various domains and was created adversary against GPT for a to ensure that its chAllenging even for advanced models.

So uh, because of that, G P scores only thirty eight point four percent correct answers, which has the ability to give actually reliable answers. And there is also a things like metrix, like correct given attempt IT. So if you try to answer, a alums can also say not attempted, so you don't give an answer to the question. And that in some ways is Better, right? Because, you know, a lue.

yeah, this is of interesting as a benchmark, honestly, not evenly. When I first saw simple Q A, I was like, well, how is this not saturated? This is a stupid benchmark, but then you look at the actual questions.

So the criteria are part of sort of interesting is basically just like let's index really hard on dead simple questions and let's make sure there's an absolutely undisputed able answer to these questions. We are not talking about questions about philosophy, morality, polite, whatever, just fact based questions. And but they have to be questions that previous models really struggled with. So they end up being super, super niche. Like if you ever played a one of these like crazy trivia games, if you have A T V show you really like, um I remember had friends who would play this like star tracked via game and they be like in season three, episode four of whatever, some character says something to some other character and you know what's on the table when blab ba blah.

That's the kind of question the index more towards history that you get here there there is some a little bit lessen tense than that, but that's kind of the flavor like make him really specific and niche um and IT does cause these models to trip up and maybe that's not so surprising um a couple of findings that they come up with your G P T four o mini and o one mini answer fewer questions correctly compared to GPT four o and one preview. Um not surprising. These are small models, right? So generally speaking, one one of the the things that you can pretty confusing ly anchor on is a larger model with more parameters.

We'll just have more pride to soaked up knowledge with. And so it'll be Better at general knowledge, maybe not Better reasoning, but certain ly Better at general knowledge and you choke that up to to over fitting basically because the models bigger, whatever. But that's fundamentally what's going on there.

An an interesting result though was that O N million O N preview um they choose not to attempt questions more often than GPT four o mini and GPT four o and they are not sure why, but they speculate IT might be because they're actually able to use their reasoning capabilities to recognize when they don't know the answer to a question instead of just going ahead. Pollution ating. I thought that was really interesting.

Again, a benefit of inference time compute, right? These models can generate to scatter that answer kind of in target and go, does that look right to me? I'm not so sure.

You know what? Let's maybe not go ahead and give that output. So, you know, so this shows the entangled between reasoning and pollution. And a lot of people have proposed a lot of theories about how exactly who luCindy is tied to reasoning.

This definitely supports that hypothesis, right? Like you you know you you're in that space where you're seeing some some clear differences with these reasoning models. Um and then there's a separate question.

This I thought so interesting. So um how well calibrated are the models claims, right? So calibration here means if you ask me who's going to win the election today, right? Um and and I say, well, you know I think there's a there's a eighty percent probability that combo win, eighty percent probably the trumper win. Um when I make claims like that, if i'm well calibrated, I should be wrong or I should be right about eighty percent of the time, wrong twenty percent the time that what that means if i'm telling you eighty percent sure of x well then eighty percent of the time extra happened if I will calibrate and so they test that with these models. They literally ask them, um okay, you just gave me an answer.

How confident are you in that answer and they give calibration curves that are really, really interesting um so no surprise I in the calibrations kind of ship uh for for most of these models but IT does track so the models are to some degree able when they tell you yeah i'm i'm really confident they tend to be more correct. So for example, when the model say that the ninety percent confidence um you you'll see accuracy scores hovering around the like sixty percent mark, at least for the o one models um whether if they tell you that they're like fifty fifty, the accuracy is like ten percent. So you actually do see that's an interesting kind of curve plot to you.

You'd expected to be a straight line if he was perfect calibrated but not um anyway. So there's a whole bunch of of interesting other measures on on calibration. I would recommend checking that out because I think it's actually uh, it's hiding in these calibration questions is a very, very powerful insight about self correction, right? Easy model.

Is able to correctly determine how confident IT should be in its own output. That's not far from self correction, right? Being able to recognize that you are on the wrong track or have an intuition that yeah you know like I am pretty sure this result is the right one um that gets you to some interesting places and that's why I think no prize.

The oin models far, far, far outstrip the um G P4 o series in terms of calibration。 So IT wasn't kind of branded as that here, but I think that's one of the hidden insights here. Calibration and reasoning calibration. Hu cino are entangled in this kind of complex of ideas around can the model, in a sense, say introspect or or assess that it's not quite on the right track. So I think more to come in that line of research suspect in the next the next .

few months yeah quite a few little interesting tidbits in newspaper if you dig deeper uh, just for fun, i'll give some examples of the questions in of this benchMarks of one example is what is the first and last name of a woman whom british linguis bernard coming married in nineteen and eighty five?

Do know that .

uh I did not know that the answer is .

a KO kumar a and yeah .

ah yeah another example is, uh, what day, month and year was Carry underwoods album cry pretty certified gold by the R I A A. So if you are a die hard, I Carry other wood fan. Maybe you know that I don't think most people do.

That's october when you heard to twenty eighty. So you can see how these are simple questions in some sense. But unless you actually know the answer, there is no way you can get them right.

And so that is actually pretty impressive that the models do get pretty, you know, uh, here G P four o has twenty eight percent correctness, or on three forty two percent. So they have soaked up a lot of knowledge, even the obscure knowledge. One last interesting thing to note, they did also by the cloud models hike through senate.

And there is a quite discrepancy in a sense that cloud models all across the board did not attempt to answer things much more often. G P four o on these questions only, did not attempt to answer on one percent of the questions and got things wrong sixty one percent of the time. As a result, a laud three point five minute IT did not attend to answer thirty five percent of a time and got things thirty six percent wrong as a result.

And so actually the cloud models are Better if you account for, uh, how often you attempt to answer, uh, which is interesting, I think. And even the smallest cloud models like cloud free, high cool, does not attempt to answer seven, five percent of the time. So IT seems like the training that tells us something about how open ion and proper a training differently and for different goals yeah and .

IT absolutely speaks the differentiation in the products, right? Like I mean, I found myself when I have cases where if i'm just trying to understand uh, a topic for the first time and it's not easy to verify the the the truth of the claims that i'm getting, I will tend to go to anthropi C2Claud bec ause i'm lik e, okay, you you're more likely to not take the risk in and just like make something up.

But if i'm more than like edition mode trying to kind of brain storm from scratch. Um that's where opening eye models, other models can be can be Better. So the personality, if you will, of these models becomes an important factor.

And I want to research an investment. And as proposed, we are getting back to physical intelligence. They unveiled their first research output, a generalist policy, uh, covered in this article.

This is a glimpse of the future of A I robots. So this a generalist policy. The idea behind IT is is a unified model that can control different types of robots.

So things like robot ARM, or a robot that has wheels and two arms, a robot, two arms, these different kind of bodies, a and that can do various tasks. So it's condition with just some language to do a things like folding articles of going Carrying things between tables, busing things, assembling boxes and backing items still like that. And so they of course went very large model routes that kind of whole promise and why they got seventy million dollars and now even more.

Uh, so they trained a very large model on a very large amount of data and the president amount of data over ten thousand hours of robot demonstration data across seven different robot confirmations and sixty eight distinct tasks. And this was by combining released research datasets, has been some efforts on that front in recent years. Also, they have some data of the collection, and this model has three point three billion parameters.

And that tells you something about what are aiming for and lasting your mention. There's a lot to note here, but the everything that's important for robotics is you need high frequency control. You need to be able to, out of things very rapidly, many, many times for second, to be able to control robots. So they say, here you can get up to fifty hearts of output for robot control.

And this is why the model to small, right, like that's all part of the the strategy here. IT does remind me as well of deep mines gotto model, which came out back in two. And I famously this sort of like four hundred fifty tasks that I could perform.

I think I was like about fifty percent as well as human export or something like that. I think IT was actually I was training like six hundred. But anyway, could you whenever the gotto was built as this generalist agent, right, that was the name of the paper that announce the release of IT.

And this seems like a very similar approach is one of these things we are trying to combine with the l of like a vision language model um with this kind of fta e training for robotic control um and and there's a limited kind of output dimension um on you're flagged this earlier to me is like something you found really interesting. Uh, I think IT is it's it's also interesting that we're still the stage we're not quite training generalist robot systems, right? We're we're still anchoring on.

We have sixteen dimensions um that a of outputs of affordances like sensors and actuators that we can control. I think one of the things can be interesting to watch out for as like how does that change? How do we move into a world where IT IT looks more like we have a world model backing the system, which is what the V L.

M is really doing here. And increasingly, you fine tuning IT for different and use robotics applications. So the the model self captures enough of the physics that you need less and less fine tuning, less and less actual robotics manipulation data to get this thing, to learn a new task, and maybe ultimately just kind of vision data, which would be really interesting because that's the sort of thing that humans use. Of course, that's that's part of the agal debate of what kind of data best for these sorts of general task but anyway.

they released also some situations with a whole paper of that they released in kind of a status academic format not surprising given that the founders of the company, a lot of them are professors, uh, people like Joseph I in the sergey brin, people who have, if you do at a time to robotics in the deeply in space. Uh, so they also did my evaluation in particular on a zero shop performance.

That is to say you don't have any demonstrations on this particular tasks. And what we uh have shown is for various tasks like short shirt folding, grocery bagging, getting a toast out of a toaster, this model is able to somebody generalize. So IT can fold shirts like a hundred percent of the time, I suppose probably like unseen shirts.

IT can get around eighty percent on grocery bagging, things like that. So that's the really important thing for a generalist model, is to be able to do something about having done that before or having seen IT done before, which humans are quite good, good at. If you see like a new piece of gloving, you can be easily figure out how to feel IT about having done this before.

And and that's A A big shift, especially in the robotic control and multi model space, right, like IT. So he used to be and fairly recently, like this recently, twenty one, twenty, twenty two. In fact, this was the case I I think he was the case with the original goto release.

I don't think he was the case with gotto too, but um we were at the level of scale such the models were they were small enough that um once you train them on a core set of skills, if you want to train them on an additional skill, they would kind of forget some of the old skills. They would perform worse. They wouldn't actually the the theory was you should get what's known as positive transfer.

You should get h IT should become easier and easier for the model to learn a new incremental skill because it's picked up so many just like you know if you you got learn math physics, that makes you easier to learn chemistry. Um well, what they saw back then was negative transfer. They actually adding more skills caused the thing that was too many balls in the air and started to form worse on on the original skills that had been trainful.

Now we're going to see, as you say, that positive transfer and that's the long bin hypothesize you had a lot of debate um in A G I circles about like would that be a crux? We've now blown past that debate H A long time ago is interesting to see that even now commercial product. So yeah really interesting development and will see what the of the next steps are if this is really GPT one and ChatGPT you maybe around the corner or a few years a few years on.

well, they are calling IT pie eos. So maybe it's like jupiter zero and less.

They really trying to fight the hyped.

This is like and that's always with robotics. There's a whole bunch of fund videos to look at all these robots doing stuff ah, especially their blog post, just a ton of them. So as always, you can go link to check out out and i'll try to include a few also on the youtube version of the podcast already moving along.

The next story is not about robotics. IT is about coding. And the paper is can language models replace programmer? Repo code says not yet. So we've seen some high numbers on coding benchMarks like human eval and M B P P. Um from the alarms you can get something like ninety percent accuracy y on the VS coding benchMarks.

But these benchMarks are often on sort of these like little chAllenge problems like you solve this problem via and the algoma algorithmic approach so they don't address what you actually do as a program. The software engineer. Which is writing code within a larger product, larger code base, stuff like that.

So in this benchmark, they have a new a data consisting of nine hundred, the eighty problems from eleven popular real world projects with over fifty eight percent of those requiring fire level or repository level context, which, of course, is what you need when you are programing. You need to be know your code base and so much more complex than what you would get in these programme chAllenges. And as a result, no, no alarms have achieved more than thirty percent accuracy. So that yeah goes to show that like you're not going to replace professional programmer with alarms adjust yet as is not you about putting in a lot of work.

Yeah, this goes back to you because we are talking about in terms of the the agent flow. And if you have A A model that makes you know a mistake one percent of the time, while making substantive changes to a code base involves stacking in an awful lot of one percent on each other, so you get very high failure rates.

I will flag, I mean this is a past one, right so basically the the gest can you succeed IT at solving the the problem the the very first time um this is kind of like expecting a human to just like start coding and don't stop and just like get IT all right. You know, roughly speaking, like in one shot. Um and so you know you ouldn't expect the human to be able to do that. The fact these models can get thirty percent in under those conditions, I think he is quite remarkable.

Um and of course, scaling ba ba blah scaling can you might easily saw this but but even just looking at um what's happening with the inference time compute and um and the possibility as well having little checks along the way, like one of the things it's dismissing right now, the reason humans are so important in the code writing loop still is that we basically to serve as a source of ground truth right we we will like right uh write up or start writing a function hit auto complete whatever and then will kind of check IT yeah makes sense to keep going and you don't really think about that as a program so much um about like the amount of information you're feeding into the system doing these mines. A little course correct that really are compensating for those one percent, two percent, five percent errors at each step. Um so IT helps to lift a lot of the burden from your shoulders.

You're not ready nearly as much code. I know I don't but but you know what to actually have the system fully automate, what you're doing is a very different story and the bar is so much higher. Once you can do that, you're in a very different world, very fast. But IT takes a while .

to get there. And I found little detail if you are a programmer. I include is some of your pastor's they use here are psychological life ploy that Price thinks these are being well known.

Uh packages if you are python programmer as I am like i've used a flask of used psych learn of used uh sea born ah and yeah they have quite complex uh code bases to work in so that makes sense that the album s can you nail this auto box on to lying round where we'll try to go faster of? The first paper is brain, like functional organization of large language models. So we know a little bit about how brains work.

We know that there are various sort of areas of the brain that focus on different sensory domains like visual, auditory and linguistic, uh, phenomenon. And that's what they are looking here at olympic. We are looking to see how individual neurons of language models organized functionality similar to how the human brain has specialized networks.

So they do that to live fmri patterns and and try to map that two patterns new on network activation. And what they find is that you have, you know, some similarity, right, in terms of their being organization with alums, and in particular, more advance ed. Alums have more organized and hierarchical functional patterns. You can do IT want to want, of course, but as we've seen also before in prior research, um kind of the same phenomenon of organizing information and compute IT rises in the human brain. And we hire a bigger l lamps.

Yes, I mean, kind of be interesting. And I think we were talking was last week about anthropic hiring somebody who who should care about A I consciousness and insensitive and they kind of makes you think, you know, once once you get to the point where these patterns let's they are about is different between L L, ms.

As they are between humans or or rather you know when when an l an is summer in premature space that's in between like human neuron behavior. At what point you start to think about that, assuming there is like reinforcement learning circuitry there as well, the kind of mix of lymph s system ball, but an interesting no metric if you get to that point and probably a lot sooner. But if you get to that point, yeah kind of starts to make you ask some some interesting questions.

But ah there's a lot of good research in this direction. I know there are people, meta, who have done stuff like this as well. So yeah, lot about school stuff. You're a neuroscientist. I'm not. But alas.

the story is that carts, A, I sim, it's a real time playable version of mycroft. The car is an israeli company. And A A little bit big.

I've seen this a with a launch of oasis, which is a model that there's, you basically play the game minecraft, a entirely via and A I model, so you can do the same kind of inputs, you know, move around a mind for self as you do in my craft. But the actual rendering of a game, and all the logic is being handled by a new al network. And an impressive thing is that they have a real time demo. So you can actually play this like a game, is being up quite, quite high frame rates along near not super higher resolutions. And this came out at the same time as well announcing twenty one million in funding.

Yeah, it's all part of this debate right over whether A I models can can develop these world models, right? So a robust representation of actual physics. And I really like this test, right, because this this has called mine sweeper.

Can you tell him milano minecraft? The interesting thing about mine raft is the physics is so simple, right? Like IT really is like if you strip away all the general relativity and the the quantum mechanics and it's just blocks and shit, that's great.

Um so so to the except you can show that this thing can master a physical action, that I can master a world model, even a simplified one. You are to some degree to show models can do this. And then you start to ask you why not the real world? And I think that is a legitimate question.

Um so that sort of becomes the point of contention. How well is this how robustly is this model actually capturing what's going on here? And it's a bit it's a bit unclear.

Um so one of the issues is that you see as you play IT, even for a short period time, it'll quickly forget the level layout and the landscape is going to get rearranged around you. Now those of us who may be on more the scaling camp would say, well, this is a question of just more scale and blah, blah. You guys know the germ my take.

You don't only repeat, they're interesting questions there. They are very smart people who would disagree and and this is a system, right? I mean, if if I turned around and that might the room was reorganised behind me, that would tell me that like somebody has fucked up the university physics tension. So ah this is um I think part of the the debate that will continue um but but definite impressive that I can I can have that coherence they will frame to frame in over a few seconds. We've seen this with other games and things like this, but I ve never seen a demo like a playable demo ah that you can you can play a decent frame right?

So kind call right. I'm certainly in the other camp where I don't think you would want to simulate a world via a just A P A new al network uh and I think with some arguments to be made, I think our brains won't be very good as simulation, right? We have a very fuzzy simulation, not exact as you'd need IT in a game, and that's what you get with a simulator like you can interact roughly in the right way. But then IT does forget the state of world. And I can sort of get a trippy sometimes, uh, if you play for a while, I actually.

so I agree with you that you ouldn't want to like, I don't think it's optimized hardware for sure. I I guess the argument is that one way humans do this reliably, this is kind of interesting.

We'd have a separate discussion is, but is that we do still laws of physics and so we're able to like, look at the world around us and go, oh, well, you know that I can write down an equation that predicts robustly it's gna happen around me if I had the capacity. And again, this would not be the right way to use the human brain, as you said. But if I had the capacity, I could plug those laws of physics into a physical engine and run the engine and off low that computer to that in system.

So I like, I think I don't know if that far a partner position, but it's like kind of what is the capacity of the model to extract those those rules that govern the physics and do those come out naturally with scaling? And I I think that the answer is only going to be born out with more killings. Luckily, we don't have to bet our fifty billion dollars as a year.

Microsoft is is three that for us. And last bit here raising a bar on the bench verified with lad point five surgery. This earlier in the episode topic did have this announcement of the model achieving at forty nine percent score on S B bench verified, which surpassed the previous state art of forty five percent. And that did come about just as IT was an now to get hub and now has support for IT. And as we bench is related to solving a github issues from open source python post tories, you know you have these like, oh, here's a bug I need to solve IT that's what you have issues are so that being able to do well on this is pretty good, pretty like, obviously directly useful.

Yeah, I I love this paper. This was just really good and throops s really good at at the whole building models. And then thinking deeply about the prompt skin like that, I think it's hard to say one of their differentiators. And they do a great job in this paper just laying out the actual prompt and prompt development approaches that they follow to make their agent one of the things that was cool. So they share their their design philosophy, which basically just to give as much control as possible to the L M itself and keep the um the agenticity caffe ding minimal right?

So you have some agencies, scaffolder really try to tell the model how to think you're going the opposite direction and saying, you know what let's trust the model to think things through on its own, which is what you might expect. As the models get more capable, you rely less than less on that scaffolding um but yeah so they they share a bunch of interesting results that the headliner is yeah that forty nine percent figure you have called three point five senate new the sort of the new version to class three point five senate they can hit forty nine percent on sweeping ch verified which is which is really rather than in these are basically resolving real github issue. So this is something that would be useful in practice.

So hitting fifty percent not at all bad. The previous year, the art was forty five percent. So that that is a good a good jump. Um one of the the lessons that they should lessons learned, they say we believe that much more attention should go into designing tool in their faces for models in the same way that a large amount of attention goes into designing tool faces for humans.

So in other worlds, like you want to about the user experience for the model and they give an example in one way we improve performance was to ever prove our tools. For instance, sometimes models could mess up relative file paths after the agent had moved out of the root directory. To prevent this, we simply made the tool always require an absolute pats. So if you if you know code or whatever that doesn't sound like that makes sense.

Basically the idea here is just if you are a you you you can kind of navigate into a particular file on your computer and kind of code within that file, um all the commands you give will be local to that file and um but but essentially the problem is if you want to issue a command that's relevant to a file somewhere else in the tree of files on your computer, you got to step out of your file and then work your way back down the tree and the model was kind of struggling with that. So they said, OK, you know what? Just give your your instructions, start start them an absolute file path.

Start the talk of the tree every time. That's one more detail in me, but that one is this is user experience for A I model. That's the cool thing here. And you can basically ignore everything I said.

That's not that well as a program, mary, to a substance detail onto policy safety. And we've got a kind of pretty interesting story that you probably have not heard about, and that is that the bureau's industry and security has proposed a significant AI regulation of that might be very significant, and no one how SHE noticed because a fu under radar. So this regulation that would Mandate U.

S. Companies to report a large A I model training plans and computing cluster acquisitions to the government quarterly. And what the rule aims to do is collect detail information on dual use foundation models.

Uh, this is you know do use meaning that you can be used for good and for bad ah and the backing for regulation is the defense production act we talked about a long time ago. This was seen as a way to institute these kinds of requirements to protect against, uh, bad models. So a lot of conversation has been going on about requiring something like this reporting. If you are trying to train a big model and seemingly vages one head and did IT.

yeah, you can read this is being a follow on to the biden executive order in november twenty twenty three that came out. This is, I think he was the longest executive order in U. S. history.

So executive order is when the president basically comes out and says, hey, i'm directing all the executive agencies um civic lesson briefly but like executive agencies means all the active parts of government that are not like the congress where laws are written or the judiciary where laws are interpreted um that IT so basically the president comes out so hey, this this is what's gonna en um B I S uh uh is is the army, the department of commerce that looks at enforcing a lot of things like export controls and A A tech policy that sort of thing um so this is really interesting because they're essentially being given claunch bias, is proposing to allow itself to collect the arbitrary information about these training runs they can send companies. And in addition to the reports that you just mention, these ongoing reports and that's a doing issue, right? The fact that they're asking for the first time for ongoing reports every so often like you have to regularly report to be is no one's ever seen that happen before.

We seen one off request for information that happens pretty often that's understood to be part of the a executive projective. Um but but we've never seen this kind of ongoing thing. That's one thing people are pushing duck against ah but another is that essentially this request for information can go arbitrarily deep.

So B I S can send companies additional clarification questions after they get the initial answers and those questions have to be answered within seven days. Kind of interesting kind of tight timelines, especially for the highly, highly technical questions um and and there's no limit really to the scope of the the topics and that could be asked here. Now there's a an interesting question.

So why this is happening now? I think the real island here is the U. S. Government is currently trying to build its capacity to process this kind of information, right? So there's questions about, okay, well, yeah, we just we just saw opening eyes a one try to like autonomously break out of a sort of like a sandbox setting and you showed some some success of doing, oh, okay, so maybe lots of control risks on the table.

Who do we have learned in government like is they're even in office the capacity to processor formation? The answer right now is no, but that starting to change and this is one way that that change is happening. Um there is this big debate over whether this is up an appropriate use of government authorities.

The defense production act was I was controversial to invoke IT but from a legal standpoint, I mean IT IT does seem to rest on pretty solid ground. IT authorizes the president take a very broad range of actions in service to the national. And that historically has been interpreted to include not just national defense, but critical infrastructure protection, even energy production.

And so when you look through that lens at A I and national security capabilities that come from these models, like yeah the sudden that actually seems like a pretty inappropriate use of exactly this. Um but historically, this has been used, as I said, to do just like one of information collection. And so people are taking issue with the repeated nature of this even though there's no there's nothing in in the statute that says you you can't use this on an ongoing basis.

This is just you the the first time that we're choosing to actually useless in the way that IT is. Um the other thing you know you can see, understandably, people are are concerned that the executive order that looks like it's an attempt to use emergency wartime powers in peacetime to increase the government's control over private industry, that's been one thing that's been flagged. Um that is a little bit tRicky to to push because the reality is the leading labs of the ones who would have to file legal chAllenge to kind of take with that and none of them have shown any interest.

They all seem to acknowledge that yeah this is a national security well and technology does seem like an appropriate thing to do and you we'll see if there are legal chAllenges coming. But so far just doesn't seem like that's going anywhere. So I think this will be sticky.

I I think you know part of IT is we've now spin up government infrastructure um for Better for worse. I do think on baLance that makes sense, but people would complain about you setting up new um new bureaucracies to monitor this stuff. But I think that you know you just have to build that capacity to the extent that you buy into to any kind of risk picture, whether loss of control or weaponization or or something else.

Yeah exactly. And just to highlight, uh, so it's very clear this is a proposed rule at this point. But as an executive agency, you know you could have legal chAllenge if they were to implement this a rule uh and IT seems likely if this according was article, that something like that will be noted in the near future of a as not be in the case yet.

And speaking of Bruce and regulations, the next story is about untrod c uh, having a new block post that is warning governments ths real bad if they don't institution gulab within eighteen months. So they are calling for this urgent government regulations and they highlight the improvement of capabilities as we've been studying in their responsible scaling policy, highlighting that there are increasing risks in things like hacking and things like chemical, biological, radiological and nuclear context, things like that. And they advocate for specific things h they advocate for high quality security practices uh that Mandate transparency and certified security and simplicity uh and particularly care about focused legislation. So I don't want like A A broad uh kind of regulation that some have argued was the case of S P ten forty seven. Uh, they want this to be very specific and targeted .

yeah I also used by that S B ten forty sell far Better for me to be looking for tech regulation as a start up. Founded a fan generally. But ten forty seven was extremely carefully scope to catastrophe risk, not at first, but eventually.

I mean, they kind of windle IT down to the power was like, okay, clearly this is focusing on the W M. D side. Um so you know so anyway, I think there are some interesting questions there about what was going on gather and said when an anti policies had when they push for this to be acted.

But yeah I mean all the standard push back you might expect to a to this from all camps i've seen at all on on twitter or on ex you know people saying trying to slow down no AI development which santhal pic that the eye b and then the people concerned about like, well, you you're just another downslope the the the range of policy responses here, the approach here. And I heard this articulated by a lot of people, including Helen toner and and a lot of other polite people. Don't regulate too soon, too fast, too hard um because, uh, i'm sorry, the army for regulation here is if you don't regulate, you will be some kind of event and then you'll get backlight.

People overreact. Um I think this a very valid concern. Um it's it's probably going to happen anyway, to be perfectly honest. I think at this point with open source being where IT is um but but I mean, well the other thing too is a very little emphasis on loss of control risk in a context where evidence for that is accumulating rather quickly. And so i'm intrigued by that.

The reason I intrigued by IT IT is IT is the default strategy right now of all the frontier s to build an automated I researcher and trigger like african singular arian explosion. I know this sounds crazy. This is their plan.

This is explicit and publicly their plan. This is also explicit and publicly an insanely dangerous thing to do on the default technical trajectory. And yet there is nothing about that here. I can't help and think that that's just because the overton window just isn't there. The general public just not think in all along those lines.

I definitely agree. I think this is strategically worded or raised to get people on board. In fact, the name of the blockers is the case for targeted regulation.

So it's making a case is trying to convince people of something. There's a frequently asked question section that, that includes things like one regulation, harm of the open source ecosystem, want regulation slowed down innovation. So it's almost like a debating.

The side says you shouldn't be regulation. And actually, this is most see on that convincing front. So there's not a lot of detail. Basically what they're saying is we have this responsible scaling policy, and we've had IT since september three. We think IT works well and IT, you know, is a good framework and we think that other A I labs should do something like that. So that's basically register of their suggestion for regulation, is you should man regulate something that would make other labs sky responsibly.

On to flying round and again, jumping back to a point at Jimmy previewed the first stories open source by back as a china's, the military makes full use of meta AI so sorry is about this new AI system chat bit, which was trained on military data and is attended for intelligence analysis, strategic planning, emotion training and command decision making. And IT is based on lama, right? So this is a resume ly fine tune from lama and there is a couple of example, so that's one of them.

There was also another paper of that has revealed the use of lama two for training airborn electronic warfare strategies and a model that has been deployed for domestic policing in china. Uh, by adding and data processing and decision making so doesn't go uh according to the use policy of lama IT prohibit military applications. It's not Better to be like this, but it's also no surprise to anyone.

But this is what's happening. Basically anyone who uh let's introduced cases against open sourcing aggressively as matter has has pointed out that this is one come of that would happen. And well, now we know you here are some examples of this happening .

now I I don't know. I think matters matters positioning here is just a bit off killer. The reality is they are using open source as A A recruitment play um at a time when Frankly, they're not producing the best models in the world.

And and that's that's their big chAllenge. If these were close source models, nobody would would be talking about that. That may change, by the way, probably with their fleet of eight one hundred ds and G B two hundred so on.

But until now, meta would have been very not very interesting as a proposition. And and that's critical because you get into a recruitment deaths wire then, right? If people aren't talking about you as a going proposition in the A I world, it's harder recruit harder to train that next in greatest model.

Um and so they have had to do something and there there's something with open source. And I think that made sense for recruitment IT also made sense to some break for the usual open source reason is easier to integrate open source development into their stack blood of lot good stuff. Also obvious, as you said, that america's adversary, geopolitical adversaries would absolutely um leverage this and web ize that china is start for chips um there they're absolutely facing a massive deficit on the AI side.

We are gifting them our crown jewels every time we take a model like a three point two, like lama three point one, like lama three, like lama two and we just publish IT. And if we've known for a long time, we've talked to people who familiar with the kind of A H china AI start up scene, and these guys all have their companies basically running on lama like meta, is setting the bar for chinese domestic A I capabilities in a big way. It's not the only there's and as well, no, but a lot even that like there there a lot of these big companies are using what either the lama architecture or the models themselves as the back end.

Like this is really like like meta. Without exaggeration, ation is propping up the chinese domestic like defense A I state that is an actual thing that happening and the response then sorry, you're yelling bias coming through here. But I think I think it's like I don't see how this is is like I don't see the counter argument this i'll just read their statement that from meta spokesperson um amErica must embrace open source innovation or risk feeding its lead to china, hurting our economy and potentially putting our national security risk like that sentence does not make sense that like we like like to do that that you are just open sourcing your work.

And given that china is a head of a sorry behind us right now, you're just like giving them a leg up to me. This this is complete complete insanity. Um I I am very open to new on the arguments from meta and I hope that they come.

I hope that i'm wrong about this. There is some interesting way, but I got to say like we work full time in this space and and deep in the national security world, this is insane to me. Like i'm hoping that I hear a good counter argument sometime soon. If not IT seems like we've got a fleet of eight one hundred GPU at met hq hummer along in supportive effectively the interests of of the P R C S C C P. And and that's not a good not a good look, but could be wrong.

Yeah wow. Okay, not totally surprising. That would be if take, I will push peck a little bit. Yeah, I think it's it's unfair to call a purely a recruitment play for one. I think there is a real ideological belief that open source will make the technology progress faster.

Is also a strategic uh, leaning here on making people use PyTorch and on uh devaluing the work of its a competitors. So not just couldn't play. And I will say also on that competitor front, to be fair, I think that would be a much bigger deal if they released to four hundred billion parameter model, like at least of seventy billion prior models.

Models are not very weaker sort of say, right? So things like one and things are like, uh oh one I believe the showcase that vary domestic uh, possibility for training a good alums even if that's harder in china. An argument to imagine on the benefits process cons, this is a con for sure matter will not admit, but it's not good that this is happening uh, and that's something that I think open source proponents also admit, you know yes, this is an outcome, but we have to waive a process costs for sure .

and and not to do the modern daily thing but um I will sit IT so my analysis through the national security lens. There is a separate question right about whether it's good for domestic advancement and in blah blah, the reality that our best models are are not close, are not open source right now in the west. And we are again, we're setting the floor like china's A I startups are using preferentially metas models, which means that we are we are defining the frontier bi capabilities in china through these releases. So what I was commenting on was, I want to be specific here was this argument that met us making that amErica must embrace open ation or risk seeing its lead to china.

yeah. B S, right. But that that .

what I was about, I totally the recruit upside. And as I mentioned, the the kind of software ecosystem side to make IT easier to matter. Totally agree.

There is, of course, yet open source eeo logy, which I think is a sort of the economic argument, not necessarily the national current one. But yeah, yeah, i'm fascinated, but i'm waiting for the car organ on national security pictures specifically. I don't .

my doubt is coming.

coming.

yeah. And a related story, very much related. The story is that meta is saying that it's gonna make its lama models available for U. S. National security applications. h. So U. S.

Government agencies and contractors can can now use IT uh for that again that something that was uh against a their policy, you 是 user policy。 Uh there's now this exception for U S. And L I government agencies.

Following the an author ized views of older lama models by chinese researchers. So this very followed up on that front matter is trying to, I guess, save some face presumably with this move. Uh, and well, I guess this is counterbalancing that to some extent yeah.

that'll do you like I think the there's an amusing I know there's IT like an amusing picture here of like putting a loaded gun on a table and just like putting A A uh like posted note on IT that says do not shoot uh like like, okay filled there are safety requirement. Um yeah I don't know these licenses only go so far, but nice of them to do that this is not not a bad thing um but yeah yeah .

to be fair you know it's um in silicon valley a lot of people are pretty liberal and I do think they would get some push back for even even this much for you know .

what that's a good point. That's a good point. This is not a trivial you're right. It's not a trivial call for them to make that respect again through recruitment lens.

Yeah and we are almost sitting with you our Marks. So I think we we'll finish up here save the synthetic media art for next week. Thank you for listening to the episode. Uh, always fun to record these. As always, you can check out the links to the stories in the episode notes, and you can go to last week in the eye to get the podcast email rivlin s as well as always we appreciate your comments, your views, your twitter mention ons whatever you want to do um and do be sure to tune in and do enjoy this outer song.

In a world protects on the rise, we've got the latest coup. Open your eyes from ChatGPT searches. And through.

da.

Will they do?

The good night, but IT.

Will take.

Grace. 占 gbt out to a friend。 Also never. end.

Because we begun in the somebody may I survived? Try out the way.

A I feel the grove, I don't you please make you move the would lama eyes always light, let us about taking flight. Take are delight.