#208 - Claude Integrations, ChatGPT Sycophancy, Leaderboard Cheats - Last Week in AI

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:11):
Hello and welcome to thelast week in AI podcast.
We can hear us chat aboutwhat's going on with ai.
As usual, in this episode, wewill be summarizing and discussing
some of last week's and maybeeven two weeks worth of AI news.
As always, you can also go to theepisode description to get to the links,
to all the stories and the timestampsso you can skip ahead if you want to.

(00:32):
I am one of your regularco-host, Andre ov.
I studied AI in grad school and nowwork at a generative AI startup.
And I'm your other host, Jeremy Harris.
I am the co-founder of Gladstone ai, AINational Security Company, blah, blah,
blah, blah, blah, blah, blah, blah.
And yeah, welcome back.
I mean, it's good to be back.
It's good to be back in the seat after.

(00:52):
God, I mean we, so we were talkingabout this earlier, but we had like two
weirdly simultaneous launches of thingsthat happened within, I wanna say, a
week, a week and a half of each other.
And so Andre was likesuper busy the first week.
Then I was busy, busy the nextweek and it's just been a, I.
Anyway, it's been a real fun time.
Yeah, the fun bit.
We are also discussing how, becausewe do this podcast, we actually

(01:16):
have to be on top of what's goingon in AI and, and not doing it.
That was actually kinda strange.
On the other hand, because it is last weekin ai, we do try to do it once a week and
it is a bummer when we have to miss some.
So we are gonna try to be consistent,at least from the next few months
until we have any more launches.
But, hopefully listeners understand.

(01:37):
Unfortunately we do have day jobsand so on, which sometimes their
priority, you know, it happens.
But, the good news is nothing hugehappened in the past couple weeks.
There's been some interestingthings to discuss and we will get
into some of those, covering somethings that are a little bit older
and some things that are brand new.

(01:58):
and that's kind of a previewof episode in tools and apps.
gonna talk about some kind of patternswe've seen with OpenAI being very,
what people call Chantic latelyand the whole drama about that.
Also some brand new newsabout philanthropic and IPC
servers, which is pretty cool.

(02:18):
Applications and business as always,a few stories about chips and China
and also some, funding news for somestartups, projects, and open source.
A few new models, and actuallysome research as well.
Research and advancements some prettyspicy, results we are gonna get into
about leaderboards and, more research,really explaining what's going on with

(02:41):
reasoning and RL and then policy andsafety, some things about malicious uses
of AI and vulnerability, things like that.
So it'll be a, a fun little episode.
I think we are gonna enjoy discussingsome of these things and jumping
straight into tools and apps.
The first story is brand new.
It's about philanthropic lettingusers connect more apps for Claude.

(03:06):
So this is basically allowing you to havedirect integration to various services.
We have a starting set ofpartnerships with things like
Atlassian, Zapier, CloudFare,Intercom Square, PayPal, and others.
The idea is that when you, to a query intoClaude, it'll have a little popup that's

(03:32):
basically like, do you gimme permissionto talk to the service at Legend or Zapier
or whatever to do whatever you want todo and it can directly do it for you.
So instead of having an AI built intoyour, I dunno, JIRA task tracker for
work that is custom, Claude can nowdirectly talk to that thing using

(03:53):
presumably this model context protocol.
Standard way to communicate toservices that philanthropic released
last year and has kind of taken off.
And it can directly talk to thatand basically be your AI for your
task tracking software, or itcan be your AI to process news.

(04:14):
It can basically now open up and be achatbot and can do all sorts of stuff.
And you know, this is similar to lettingyour AI just do web surfing for you to
do whatever you know, it needs to, tofulfill your task, but I guess much more
elegant and direct where it can talkdirectly to the service, it can query

(04:36):
it for you without having to do the,I dunno, like grunt work of pressing
buttons and logging in and so on.
So I think pretty exciting in terms of arelease for Claude that really makes it
much more broadly useful and, and kindof impressive to see them taking the lead

(04:56):
in this particular way of using chatbots.
Yeah, it definitely seems like Anthropicbuilding on the early advantage they
had with the MCP protocol, whichOpenAI obviously has since taken on
board and, and other companies too.
So it is becoming the defactostandard and it positions
anthropic really well in the space.
It's also, I mean, it's consistent withthis vision, right, that we heard well

(05:18):
many times, but kind of most famouslyarticulated in that Leopold Ashkin
burner situational awareness thingabout the drop in remote worker, right?
This is really a step in that direction.
You've got a model now able tojust call these tools directly.
It's being productized, it is beingrolled out this version at least to
Claude Max subscribers enterpriseplan subscribers and soon to pro.

(05:42):
So again, this is philanthropic,kind of finding the sweet spot of
what they're going to charge for thekind of higher tier subscriptions.
That's been a questionrecently too, right?
When they introduced Claude Max theysaid we would give early access to
people who sign up for that tierearly access to new capabilities.
This is apparently one of thosecapabilities they flagged for that.
So starting to kind offlex that muscle a bit too.

(06:04):
But yeah, this is, I mean, this is on thestep to fully replacing certain kinds of
of like well, it de depends on the, the,the way you wire things up, but certain
kinds of engineers, certain kinds of well,it de again, if you're doing some kind
of like sales backend work or whatever,there's a lot of stuff that could be
straight up automated down the road ifthey keep pushing in this direction.

(06:27):
So kind of interesting and we'll seewhat the impact is too on the job market.
I mean, there are some indications thatthis stuff is really starting to rattle,
especially juniors or entry level roles.
But yeah, well it definitely a,a big cost savings if you're able
to, you know, get these sorts ofagents to do your work for you.
Exactly.
I know personally, you know, as someonewho does programming so far you've had

(06:50):
to sort of wire out things yourself.
Like let's say you want to writea script to process a spreadsheet
to do some work for you.
Typically that's involved writing ascript to really do it efficiently,
you know, to not have to downloadit, attach it, write the prompt.
Now it makes it much easier to automatethings via prompt because you don't need

(07:11):
to do any sort of manual steps whereI can directly talk to whatever data
source it needs to, to do the task.
So a simple example, again, just to makethis clear, is they show you being able
to ask what's on my calendar, and thencloud can directly talk to your calendar.
You have to press a little button toallow it to get the data, and then it

(07:32):
can answer your questions about that.
So really, I do think kind of a,a pretty significant step in terms
of expanding the capabilities ofLLMs and this kind of service to do
all sorts of stuff for you in a waythat you could not have done before.
Worth noting Also, as far as newfeatures goes, they did launch their own

(07:53):
research tool because apparently everysingle provider of LMS needs one and
they are launching an advanced researchtool, which is their fancier one.
It can take five to 45 minutes tocompile comprehensive reports for you.
So also interesting to me that it turnedout for AI and for these reasoning models,

(08:15):
that deep research has turned out to beone of the, I dunno, power use cases.
And next up we are gonna talkabout open ai and they've had
something pretty embarrassing, Iwill say, in the last couple weeks.
So if you're on Twitter, if, or ifeven you use Chad GPT, there's been a
lot of discussion of a recent updateof g PT four oh, where they have made

(08:40):
it, let's say, very enthusiastic andpositive when communicating people, I
didn't know this word, actually, glazingapparently is what people describe it as.
Yeah, yeah.
Where yeah, basically the model islike, you enter a basic query or
something, or something like that, andthe model just cheers you on to no end.

(09:02):
It's, it's sort of crazy, youknow, telling you, oh, this is
such a deep insight, this is sucha good idea, et cetera, et cetera.
And it was so bad, and there's beensuch kind of bad examples that OpenAI
seemingly really rushed to fix it.
Sam Altman actually announced on x thatthey are working on some fixes, ASAP

(09:25):
to address the personality issues fromthe last couple of GP four updates.
They, rolled out update to the systemprompt that some people have talked about.
They've also seemingly done a fullrollback of GPT to a previous state of it.
So.
I would say, you know, there'squestions as to how this happened.

(09:47):
It's potentially the case that they tryto make it overly optimized for engagement
or for positive feedback by users.
But it's clearly like when you lookat some of these responses, it's clear
that something went wrong here and it's,it's something we haven't seen from one
of the major players yet in this way.

(10:10):
It's also hard not to notice that thisis happening just weeks after OpenAI
announced that they're no longergoing to be focusing on persuasion
capabilities as part of their preparednessframework in the same way as they had.
So when you think about persuasioncapabilities, certainly syco fancy
in these models is something that youmight correlate with persuasion, right?
Telling people, oh, you're,you know, you're so smart.

(10:31):
What a great idea, what a greatquestion when you optimize.
And I haven't seen clearindications that they had
optimized directly for, for awards.
I, I've seen some posts on X ofpeople saying like, Hey, here's
a, an option that showed up.
You know, do you likethis personality or not?
Like thumbs up, thumbs down type thing,which to be clear, I think is a gigantic

(10:54):
mistake, a really, really dangerousprecedent for OpenAI to be setting.
Frankly, I mean, we've seenOpenAI do related things.
Be willing to kind of pushthe envelope on some stuff.
You could often argue for it or whatever,but when it comes to like optimizing,
when you close that feedback loopbetween yourself and like the, if you
will, the person that you're talkingto, to make them more agreeable or more

(11:18):
likable by you, I think that is prettyclearly a very, very dangerous thing to
be doing when you have as much computeas they do when we already have evals
that are showing these models are reallyeffective at persuasion and manipulation.
That's the sort of thingyou start to think about.
At the next beat of scale, at thenext beat of sort of subtlety and
persuasion and manipulation, whichthese models seem to be on track for.

(11:40):
So anyway, I think this isdefinitely a space to watch.
There's not necessarily going to besmoke the next time there's fire.
And that's something that I thinkpeople really need to understand.
These models are, by definition, gettinggood at persuasion means, or almost by
definition, it means that the other persondoesn't realize that's what's going on.
So as you keep pushing in that direction,as you use more and more subtle cues,

(12:01):
feedback cues from users, I think a lotof people have very justified concerns
that we're heading in a direction where,you know, there's a certain amount of
asymmetry between the user and the companyhere, where the company is able to think
on computer clock time about how tooptimize their relationship with the user.
That's not necessarily healthy, especiallyaggregated over the entire population.
You know, hundreds of millions ofpeople interacting with this stuff.

(12:23):
Right.
And just to get into some basicexamples, a lot of this was kind
of funny and people started postingexamples where they directly got
to LM to be as silly as possible.
So one example, just pulling off ofTwitter, someone says, I just woke
up, did two pushups, and my brushmy teeth in the next six hours.

(12:44):
Cha Petti said, you have achieved a levelof mastery if you dare to even imagine.
The delicate art of strategic patienceto Ving is itself a small revolution to
do two pushups immediately afterward.
Is it a curation of war against inertia?
I will say perhaps this exampleis you know, tweaked, I'm

(13:05):
just pulling off of internet.
But that shows you kind of theflavor of what you're seeing.
It's of a model is being very mucha suck up, saying very extremely
positive things that are not natural.
And I just actually searched andopening, I just posted a blog post today

(13:27):
as we are recording titled, expandingon What We Missed with Sika Fancy.
And they go into, you know, in April25, they pushed an update, update, had
a few things, each thing individuallydidn't look so bad and their metrics
were good, et cetera, et cetera.
We are talking about.
What will improve in ourprocess, what we are learning.

(13:49):
So a pretty embarrassing kindof situation here, right?
The fact that they needto address it so strongly.
Some people also compared it.
I remember to the Gemini launch fromGoogle where there were very silly
things going on with the image generator.
Yeah.
I think OpenAI for the firsttime has, has really fallen

(14:13):
on its face with this launch.
And as you said, there are some realdangers to doing this kind of thing.
Another thing that people pointed outis some people are getting very close
to these tragedy g BT models, peoplewho are perhaps possibly delusional

(14:33):
or in a bad mental health situation.
You know, talking to the chatbots can seriously affect them.
And so you need to be careful with howpositive, how affirming chat visha bots
can be and how, you know, how much theyreinforce whatever you're telling it.
That has real implications, evenaside from, let's say, theoreticals

(14:55):
of per persuasion or things like that.
So, yeah, a lot of discussion Ithink will be going from this event
and, and some studies and so onto really get into how you can tip
models to be a little bit extreme.
And otherwise quite aninteresting phenomena.

(15:16):
A few more stories.
Next up, we have a newmodel launch from Baidu.
They are announcing Ernie Xone and X five Turbo, X five.
Turbo, as you might imagine,is the fast kind of model.
They are saying that it has 80% pricereduction compared to its predecessor.

(15:42):
Ernie X one is the deepreasoning task model.
They're saying it's betterthan deep seq, R one and O one.
Things like, you know, deep chainof thought, things like that.
So, Baidu and as one of the leadingcreators of LMS are in China is,

(16:03):
you know, really, I, I don't knowif it's fair to say catching up,
but keeping up with what's goingon with philanthropic and OpenAI.
You know, increasingly you havesmall, cheap, fast models like Gemini
2.5 Pro or let's say O three Mini.
And you have these quite big, quiteexpensive models like O three like

(16:25):
Cloud Opus, Gemini 2.5 Pro, whichare more and more very capable.
And that seems to be thecase with these two models.
Yeah, I mean, don't,don't count out China.
And, and I think there, there arereasons and we, I'm not sure if
we're gonna talk about them todayexplicitly, I'm trying to remember.
But there are reasons to, to expect thisto continue at least into next year by

(16:48):
which time the sort of chip export controlstuff is gonna have more of an effect.
but for right now, expect Chinafrankly to, to do damn well and,
and quite possibly catch up fullyto the frontier of Western ai.
I mean, that's a concerning thingto be saying, but that is the trend.
I think until, yeah, until we get thenext generation of data centers online

(17:10):
we're not gonna see that significanta gap between those two groups.
yeah, the benchmarkslook really solid here.
I mean, you know, they, they look atvarious, any multimodal benchmarks
for 4.5 turbo and certainly that'swell in advance of gPT-4 0.0
and competitive with GPT-4 0.1.
In fact, beating it at manymultimodal benchmarks, that that

(17:32):
is a, a pretty noteworthy thing.
and competitive pricing as well.
I mean, you mentioned, you know, ErnieX one turbo is like something like, was
it 25% I think they said of, of R one?
in pricing.
So it's pretty, like,that's pretty damn good.
Also, I mean, again, Rone is an oldish model.
It's an oldish model.

(17:52):
It's been around for literally weeks,guys, it's been around for weeks.
It's, it was the start of a year, youknow, that's when all this reasoning
stuff kicked off feels like forever ago.
A hundred percent.
But, but because of that, thereis so much low hanging fruit right
now in the inference stack that,yeah, like you can learn a ton
of lessons from looking at R one.

(18:12):
A lot of these models, by the way, distilloff of R one and you can kind of tell in
there thought traces end up coming out.
There's some, some similaritiesthat look suspiciously similar.
I, I don't know if that's the casefor Ernie 4.5, I haven't actually
checked that one, but we'll talkabout a model a little bit later.
A Chinese model actually thatsort of has that characteristic.
So there's a lot of ways in whichI. You can build off of R one, both

(18:34):
by distilling data directly from it,but also just by learning lessons,
infrastructure lessons and, and, andarchitectural lessons from it that allow
you to drive down that pricing a lot.
And anytime there's a new paradigm thatgets discovered or invented, you have
a rapid improvement in a lot of the topline metrics just as people find all that
sweet, low, low hanging fruit associatedwith that a new kind of paradigm.

(18:57):
So that's the phasethat we're in right now.
Expect these prices to, to kind ofcollapse faster than the traditional
pre-training kind of base model pricing.
Currently is, you know, think back tolike how quickly gpt three's pricing
dropped, for example, or chat GT'spricing dropped in the early days.
That's what we're seeingright now as well.
And, and those other prices continue todrop, by the way, even for base models,

(19:18):
but we're just in this unusual kind ofvery rapid acceleration in that in that
phase where we're getting efficiencygains that are really, really rapid.
Yeah, I remember when model pricingused to be per thousand tokens, and
then at some point they switchedover to per million tokens.
That's a good point, right?
I, yeah, I, it's funny, I don't thinkI ever consciously registered that.

(19:39):
I was just like, yeah, of course.
You know, of course we're bumpingit up by three orders of magnitude.
And next, moving away from LLMsfor a bit towards image models.
The next story is about Adobe addingmore image generators to their services.
So they're launching Firefly imagemodel for and Firefly image model

(19:59):
for Ultra with some other updates.
So image model four is meant tobe faster and more efficient and
offers up to 2K resolution images.
Firefly Image Model four Ultra isfocused on red rendering complex scenes
of more detail and REALI realism.
These are now available in the Fireflyweb app, which also has their text

(20:23):
to video, text to vector stuff, andthey're introducing this new thing
called Firefly Boards, a collaborativegenerative AI mood boarding app in
public beta, so that's kind of cute.
Last up.
They're also now adding support to thirdparty AI models like the GPT image model.

(20:44):
Google's image free Google's viatwo for video and other third
party things as well, which Ithink is, is kind of notable.
If you are thinking that, you know,this can be this service to use for
image generation for experimentationhaving third party support is
not kind of a trivial detail.

(21:05):
They actually emphasize that these thirdparty model are for experimentation
and marks their own models as quote,commercially safe, which is yeah,
highlighting what they are arguing is thereason to stick to the Firefly models.
The fact that they've trained it on noncapari data, you're not gonna get any

(21:27):
sort of trouble with using Adobe's models.
Yeah, first of all, I mean, it makesall the sense in the world, right?
In a world where all thesemodels are becoming commoditized.
I mean, this is really the ultimateexpression of the commoditization of
these image generation models, right?
You literally are a click awayfrom using the alternative, right?
For so, so it's greatfor the, the customer.

(21:49):
It's also, it makes it so that the actualvalue in the value chain plausibly is
no longer gonna be concentrated withthe model developers, at least for
text to image or things like this.
Instead, it.
Well, it'll shift somewhere else.
Obviously the hardware stack, I mean,we've talked a lot about that, especially
in the last kind of two years, that that'swhere, you know, the NVIDIAs of the world.

(22:11):
Maybe the AMDs, the as mls, the tsmcare kind of where a lot of the value in
the value chain ends up being captured.
But there's also theaggregation point, right?
So Adobe making a play here to becomean aggregator of sorts of these
models, definitely a, a good play.
Also with them leading the way on thewhole idea of in, you know, indemnifying
users, if it turns out that there'sa, a copyright violation or, or a sort

(22:34):
of claimed alleged copyright violationfrom the image generation process, not
necessarily being able to guarantee thesame thing for the other models they host
on their platform, which is where they're,they're sort of flag there for like,
Hey, you know, our thing is, is businesssafe, the others are for experimentation.
That's kind of where that's coming from.
A sort of nice way toencourage people to use theirs.

(22:54):
Now I think a lot of these companies.
Have similar sort of indemnificationguarantees, so it's not actually
clear to me that there is a materialdifference in all cases relative to
the, the promises that Adobe is making.
but I'm not sure having not gone throughthe specific list of like all these,
these models, there may well be somethat, that don't offer indemnification.
So still interesting Adobe makinga good play and, and these, I mean,

(23:16):
these models look really good.
Like they, they have some examplesand, you know, I, I keep saying
this, every time there's a new imagegeneration model, I'm like, I don't,
I'm at the point where I can't tell thedifference between subsequent releases.
Maybe it's just the prompts that theypicked here, but that they do seem very
photorealistic and, and compelling.
So anyway, seems overalllike an interesting move.
Very strategic shift for Adobe for sure.

(23:37):
And one of the few things that Ithink they could do to make sure that
they're still relevant in the long runif they don't have access to the kind
of compute that their competitors do.
Yeah, and I think the fact that they'reinvesting a lot in this Firefly web app
is interesting in a sense that they dohave an advantage in this competition.
Similar to Google in a way, in that, youknow, if you're already paying for Google

(24:00):
Workspace, you're maybe gonna use Gemini.
If you're paying for Microsoft 365,you're maybe gonna use copilot.
If you're paying for Adobe tools and theydo bundle their tools in a subscription,
you know, for Photoshop or photo editingor whatever, they can bundle in the
AI and then push you towards usingFirefly and not some one of many other

(24:25):
services you can use to generate images.
So I could see Adobe really making itout just by being the default for a
lot of this kind of professional work.
And speaking of image generation,next story is that OpenAI has
made their upgraded image generagenerator available to developers.
So we saw in late March the launchof the, what I think they call Chad,

(24:50):
GPT image generation GPT Image one.
And for a while you can onlyuse it via the web interface.
Now you can use it via the API.
And this is quite notable becausethis model does have some very real
advantages over previous models.
It's much better at editing imagesgiven an image and a description.

(25:13):
It is very good at very kindof clean edits that previously
would've been very hard.
These images are watermarked with madedata and you can kind of track it there
being AI generated, things like that.
So I think currently few otherservices provide this level of image

(25:35):
editing and I would be curious tosee, I guess, what impact this has.
Pricing is also like, it's non-trivial,it's 2 cents for a approximately 2 cents
for a low quality image, approximately19 cents for a high quality square image.
So, you know, if you think aboutthat, like that's a buck every five
images is, it's, it's not nothing.

(25:57):
But anyway, obviously that'llcollapse in, in price pretty soon too.
But yeah.
kind of cool consistent shift to ohman, I'm trying to remember who it was.
I think it was Steve Bomber, right?
With that famous up on stage ofMicrosoft, like clapping his hands
going, developers, developers, developersdevelop, well, this is that, right?
Everybody's kind ofmoving in that direction.
It's increasingly a matter of, and thisis like opening eyes, like original play

(26:21):
back when G PT three I think came out.
They were very much in that mode ofsaying, look, we're just gonna put
everything in developers' hands, seewhat they build with our stuff rather
than necessarily like the implied claimwas, rather than necessarily doing the
Amazon thing where we actually startto notice which products are doing
really well, and then we offer theAmazon Basics version of that product.

(26:42):
And eventually that's bad for peoplewho use the platform merchants.
Opening Eye has done some of that.
There's no question.
I mean, that's part of what itmeans to be in the image generation
business, but more, more APIs, right?
Like that's very a, a veryopen AI thing, and it's a very
well industry thing now, right?
That's where everything's going.
And last, sorry.
For section dealing with X AIand being able to see things

(27:08):
as opposed to make images.
They have launched GRvision in their IOS app.
So as we've seen demoed many times,you can point it at something
and ask it questions aboutwhatever you're pointing it at.
They're also launching some otherthings like multilingual conversations,

(27:28):
realtime search in voice mode.
This is available to Android userson the $30 per month Super Rock plan.
So still, yeah, XAI rapidly in catchupmode with, in this case, I guess it's
the advanced voice mode from chat,GPT where you're able to ask questions

(27:49):
about, I dunno, equations and stufflike that as open, I demoed last year.
Yeah.
I continue to be impressed athow fast GR is getting stood up.
I mean, just the sheer numberof like, they're, they're not
supposed to be, a massive contender.
They've been around for all oflike, what, two years, 18 months?
And yeah.
Already pumping out reasoning models,multimodal models and all that.

(28:11):
So yeah, they've definitely, they'retaking advantage now increasingly too,
of their partnership with X or theirintegration with X. So we'll I guess
see that reflected more and more too.
Yeah, in the very rapidly rollingout, I guess what seems to be more
and more of a basic set of featureson the chat bots, things like canvas
search memory, you name it, whatever,you know, Chad G, PT or Claude have

(28:36):
introduced over the last couple years.
Grok is rapidly adding it as well
And onto applications and business.
First up, we're gonna talk aboutthe startup from Mia ti, the former
CTO of OpenAI, who left after thehigh profile disagreements with Sam

(28:58):
Altman being asked in late 2023.
Mia Mirati left, I believe in kindof 2024, maybe around mid 2024.
We've known, she's been working on thestartup called Thinking Machines Lab
for a while, and now we are gettingsome news about their fundraising.
Apparently they're raising 2billion at a $10 billion valuation.

(29:22):
And the interesting thing thathas come out of this is that
Mira Mirati will have a unusualamount of control in this startup.
So basically what it sounds likeis she will always have a majority
on any major decision in, let'ssay the board, for instance.

(29:45):
So even if she installs hostile board,for instance, and they all disagree with
her, my understanding is she'll be ableto override and have ultimate decision
making capability as the CEO, which is.
Unusual.
It's, it's usually, you know, theCEO has a lot of power, but not
necessarily a codified majoritydecision making power from the outset.

(30:12):
So yeah, I mean, it is been kind of aslow rollout for fingering machines lab.
It's been a bit quiet as towhat we're doing, but they have
been recruiting and seemingly, Iguess getting investors on board.
Yeah, I mean, their rosteris absolutely stacked.
You know, Alex Radford famously will bedoing at least some advising with them.

(30:34):
A whole bunch of the, the kind of posttraining guys from OpenAI and well as
john Schulman formerly from OpenAI,then formerly from Anthropic, one of the
co-founders of OpenAI, in fact jumpingship and then going to thinking machines.
Something interesting is happening there.
I mean, there's no question thatlevel of talent flocking to, to
that company is very interesting.
Also interesting to see thissort of consolidation of power.

(30:55):
This is something that allthese rockstar employees.
Are actually perfectly happy with, right?
So there is this super votingmajority that Mira has.
Apparently the way it's set upis her vote on the board has the
equivalent force of the vote ofall other board members plus one.
So functionally, there isn't aboard, there isn't board oversight.
That's what that means.

(31:16):
is by the way the function of the board isbasically to hire and fire the CEO right?
To hold the CEO accountable.
That's the whole idea behind a board.
So the fact that that's nothere is very interesting.
It means she's got anawful lot of leverage.
So she's raised ostensibly about $2billion at a $10 billion valuation.
Andreesen Horowitz is in on thoserounds, and they're like, you know,

(31:38):
famously very founder friendly,allowing her to, to do this.
That's also true, by the way,at the level of the shares.
So just to give you, like, if you're nottracking the whole corporate structure
set up, typically you have a board thatcan hire and fire the CEO, and then you
have the shareholders of the company whocan sort of swap board members around.
That's usually how things work.
And even at the level of the shareholdersMira also ha, ha has or enjoys a lot of

(32:02):
control, very unusual amount of controlthe, the startups founding teams.
So some of these elite researchers who'vecome over from OpenAI, from philanthropic
and elsewhere have apparently supervoting shares that carry a hundred
times as many votes as normal shares.
And they've agreed to let mi up.
Vote for them by proxy.
So that's a lot of, that's alot of power that she's got.

(32:24):
You know, on the shareholder side,on the board side and as a CEO as
well, everything I've heard aboutMira does seem to be quite positive.
Interestingly.
So some of the former open AI employeeswho've been through the whole board
coup fiasco thing had pretty damnpositive things to say about her.
I thought that was kind of interesting.
I've, I've never met her myself,but it was in the context

(32:44):
of what happened with Sam.
She was sort of left in the lurch, youknow, back then when the board sort
of refused to tell her that the reasonthat they had fired Sam was the evidence
that she herself had provided that'snow public, that that was the case.
But without telling her thatshe was kind of left in lurched.
So anyway, she's, she's definitelyexperienced at navigating a lot
of board drama that maybe what'sreflected here in this move.

(33:06):
But it is highly unusual, and again,this would only happen if she had
an extreme amount of leverage overthe investors who are coming in.
That doesn't mean by the waythat it doesn't get refactored
at the next fundraising round.
You could easily have investors whocome in and say, look, I'll give
you the 20 billion you're askingfor, but you're gonna have to do
something about this board setup.
We want some measure of, ofreal and effective control.

(33:27):
And so, you know, all these thingsare to some degree, temporary.
But for right now, with the 2billion that they're apparently
raising that's, this is gonna bethe lay of land for a little while.
Next up some chip talk and we'vegot a couple stories about Huawei.
So one story is discussing the Huaweinine C and basically just we've already

(33:51):
discussed, I believe this chip, it's acombination of two nine 10 B chips that
combined are about as good as the H 100,not the, you know, top of the line at
the chip, but what used to be top of theline for Nvidia a couple years behind.

(34:11):
And the story here is justsaying that they are getting
close to starting mass shipmentspotentially as soon as next month.
Another story is also saying thatthey are working on a new ship that
is called the Ascend nine 10 D. It isin the early stages of development.

(34:33):
It'll require testing and thiswill be the chip that is gonna
be more powerful than the H 100.
Potentially could be, youknow, the default if export
controls get tighter on Nvidia.
As is very possible at this point.
there's a lot to be said here.
I think the, the top line needsto be a recognition that us export

(34:55):
controls actually have been working.
They just take a long time becauseof the supply chain dynamics.
China has enjoyed the ability tobasically black market import a whole
bunch of chips, h twenties h eighthundreds, h one hundreds that they
shouldn't have been able to import.
That's what's reflected unambiguously insome of the latest big runs that we've

(35:17):
seen the sort of post deep seek era stuff.
So I, I think that's really important.
China will be trying to convince us thatthe export controls are not working.
We know they are because we've heardit from literally like the founders
of Deep Seek back in the day beforethe CCP was watching their every move.
Now their tone has changed, but thefact remains anyway, so we are gonna

(35:39):
see this chip is gonna be slower.
Th this is the nine 10 D so this kind ofnext generation will be slower than the B
Series Blackwell series of Nvidia chips.
There are reasons though to suspect thatthat may not be the deciding factor.
So what China's really good at is takingrelatively shitty GPUs and finding ways

(35:59):
to network them together to make systemsthat are just really, really powerful.
Even if the individual chips within themare kind of crappy, the trade off that
they end up making is because they can'tuse the exquisite like three and five
nanometer and four nanometer nodes atTSMC to, to fab these things down to crazy
high accuracy because they can't use that.

(36:22):
They can't have chips that are asperformant on a per watt basis.
So they have chips that are significantlyless energy efficient, but that
matters less because in China,energy is much less of a bottleneck.
They're putting nuclear power, like inthe last 10 years, they have added an
entire America worth of power and likethe whole US electric power output, they

(36:44):
have added that in the last decade in theform of nuclear and, and other things.
They can actually bring nuclearplants online really quickly.
'cause they didn't go through thisweird phase where, you know, America had
an allergy to nuclear and, and so nowthey're in this beautiful position where.
Yeah, the US has export controlson these high-end chips.
The, and anything fromTSMC above a certain node.

(37:05):
But the reality is China doesn'tcare as much because they have
so much domestic power available.
So they'll use chips that are lessperformant on a per watt basis.
And you know, what's the difference?
We've got 10 gigawatts of sparepower around three gor dem.
Let's just throw it at this, right?
So that's kinda what we're seeing.
The, the calculus, the design calculus,if you're Huawei, just looks different.

(37:26):
It looks more like let's crank as manyflops as we can out without worrying
quite so much about the power consumptionand, and let's make it up in networking.
Let's make it up in the backend,in the scale up in, the fabric that
connects all these different GPUstogether at the rack level and beyond.
And that's really what we're seeing here.
And so it's this weird combination ofthey are getting some of the high end

(37:46):
chips because we've done a shit job on ourexport controls, which we need to improve.
But then there's also, they can be abit sloppier at the chip level as long
as they are exquisitely good at thescale up kind of network level, which
is what they did in particular, whatthey did with the Cloud Matrix 3 84
system that I think we talked about.
Maybe a couple weeks back, but thisis like the ultimate expression of

(38:07):
like how you wire up a bunch of thesenine, 10 C processors to beat systems
like NVIDIA's, GB 200, the NVL 72,which is like the top tier right now.
Just in, in just think of itas like brute force, right?
Like we're just gonna hook moreof these things together and who
cares about performance per whatjust because we can afford it.
Yep.
And this is following up on in earlyApril, the US did introduce new export

(38:34):
control that seemed to limit theexport of the H 20, the GPU that was
specifically designed for selling to Chinabased around previous export controls.
And Hui also announced the Ascend nine20, in addition to this nine 10 C nine

(38:56):
10 D, which is more comparable to age 20.
And the reactions to the announcementsof the nine 10 C were very dramatic.
Nvidia shares dropped 5%.
5.5%. A MD fell morethan free, broad fell 4%.
So this is a big deal for Nvidia,for, GPU space in general.

(39:20):
Yeah, it's the Nvidia thingis interesting, right?
'cause you, you might nominallythink, well, NVIDIA's revenue,
16% of it is currently from China.
It's a bit less now.
So it's, you know, not such a big deal.
You expect 'em to sortof grow out of that.
But the argument Nvidia is making,and in particular that they're making
of the White House, is you are givingChina the opportunity to refine,

(39:40):
it's to increase domestic demand,obviously for Chinese GPUs because we're
preventing them from importing our own.
And ultimately that may lead to ChineseGPUs competing successfully with Nvidia on
the global market which would then wrestlemarket share away from Nvidia there too.
So that's part of what the market seemsto be pricing in here though for various

(40:03):
reasons, I think that is very overblownNVIDIA's own earnings calls, like suggests
that they don't think that it's quitesuch an issue, at least historically.
And so there's thatinteresting dynamic too.
And speaking of the Chinese market andexport restriction, we also have a story

(40:23):
of ance, Alibaba and Tencent stockpilingbillions worth of Nvidia chips.
This is sort of an overview articlesaying that these leading internet
companies have accumulated billionsworth of the age 20 chips prior to the
cutoff of the shipments of these things.

(40:45):
In April, I think we coveredanother story to adverse effect, you
know, pretty much another, I guessoutcome related to export controls.
I mean, look, this is like Logic 1 0 1.
You tell you, you telegraph to youradversary that you're gonna bring
in export controls on a certainproduct that they need desperately
for a critical supply chain.

(41:07):
And your adversary obviously is gonna go,okay, I'm gonna start stockpiling this.
Like, I'm gonna start getting as much ofthis shit in into my borders as I possibly
can before the export controls hit.
You know, we've seen this with multiple.
Rollouts.
We saw, saw this with the A 100.
We saw this with the H 800.
We've seen this with the H 20.
We've seen it with highbandwidth memory, like over and

(41:28):
over and over and over again.
We have to learn this stupidlesson that we never should have
had to learn in the first place.
That when you fucking tell youradversary you're going to close a door,
they're going to try to get as muchshit through that door as they can.
So, like generally, if you're gonnado export controls, do 'em hard, do
'em fast, do 'em without warning.
One of the perverse incentives thiscreates, by the way, is Nvidia.

(41:50):
If they know that the door is gonnaclose on the Chinese market when it
comes to age twenties, well have anincentive to prioritize shipping those
GPUs to the Chinese market over Americancompanies because they know the American
companies are always gonna be there.
The Chinese ones won't be, atleast for this class of product.
And so, yeah, you're literally causing oneof your biggest companies to essentially

(42:13):
turn into a proxy arm of your adversaryfor the purpose of kind of getting stuff
out the door before the gate closes.
I got a lot of issues with exportcontrols and the way they've
been managed historically.
This is something that, fortunately Ithink there's a lot of investment that
the government's about to make in the BISthis is the bureau at the Department of
Commerce that does export control stuff.

(42:34):
They need a lot more teeth and a lotmore staffing to be able to, to do this.
They've been ahead of the curvein many ways, but like without
the resources to actually dostuff on a fast enough cadence.
So anyway, this is like$12 billion in rush orders.
By the way, $12 billion in rush rushorders around a million age twenties.
That is like a full year's supply.
That they tried to getin by the end of May.

(42:56):
The actual number that was delivered,by the way, did, did fall short.
Because the, the administrationannounced in early April that the
chips would need a license for export.
That was not expected.
They were sort offlip-flopping back and forth.
But to give you an idea of how profoundlyunsurprised the Chinese ecosystem was here
this is a, a quote from an executive witha supplier to bike dance and Alibaba, who

(43:18):
was involved in a lot of this shipping.
He said the Chinese clients are very calm.
They knew it was coming, and theyhave been prepared for this day.
They told us that their aggressivegoal to build more data centers
this year remains unchanged.
So their entire plan forthe year is unaffected.
Like they're moving along, likeit's business as usual after we've

(43:38):
just supposedly closed down likehard on these export controls.
So this is the kind of thing likethinking one step ahead logic that
we really need to get better at.
This is, unfortunately, it's a functionin large part of, you know, BIS
being historically just understaffed.
And again, hopefully somethingthat's gonna change soon.
But the, yeah, big issuefor us national security.

(43:59):
And one more story in a section dealingwith GPUs and hardware, there is
speculation and, and rumors and some,I dunno, reports that Elon Musk is
trying to raise tens of billions ofdollars for xai with a plan to build
Colossus two, the I guess SQL to thecurrent massive supercomputer that

(44:24):
has 200,000 VD GPUs, Colossus tworeportedly will have 1 million GPUs.
And to give you perspective, just thecost of buying 1 million Nvidia GPUs could
be between 50,000,000,060 $2 billion.
And that's not even counting, youknow, infrastructure, things like that.

(44:48):
If you add it all up, presumably it'sgonna take, I don't know, a hundred
billion, something like that to build adata center, a supercomputer Elvis scale.
A neo musk is trying to raise10 billion, tens of billions
of dollars for this simulate.
Yeah, I mean, it, it, it's kindof wild when you think about it.
The US is a $20 trillion economyand we're talking about pouring

(45:11):
hundreds of billions of dollars intothese data center builds for 2027.
That's like, we're getting to thepoint where it's on the order of like.
A percent of like the entire US GDPthat is going like, that's insane.
That's insane.
This is either the.
Most enormous waste of capital thathas ever happened, or, hey, maybe these

(45:35):
guys see something that we don't, youknow, like the idea of the returns.
I mean, they've gotta find a way toactually make back a hundred to $125
billion from these sorts of investments.
That's just one company.
And you've got, you know, Microsoft,you've got Google, these guys are throwing
around, you know, 80, a hundred billiondollars a year on their AI infrastructure.
Buildouts, this is like multipleaircraft carriers every year

(45:58):
that they're just throwing down.
So I guess it's a, an open challenge to,you know, if you think you know better
than these companies, maybe, maybe.
But it's looking pretty likelythat something interesting is at
least they see something reallyinteresting happening here.
Yeah, so he's quoted apparently,as having said that we are going
to quote, put a proper value onthe company in reference to XAI.

(46:19):
And people apparently on this call, tookthat to mean and this is just speculative
that they will have a very large raise.
And speculation is on the order oflike, you know, $25 billion on maybe
150 to 200 billion, all speculation.
But that is apparently the kind ofconversation that is going on right now.
So, yep.
Wouldn't be, wouldn't be too shocking.

(46:40):
But this is what it means, by theway, when we say a gigawatt, right?
A site for a gigawatt of power.
You're talking on the order of amillion GPUs, and there's like.
There's a lot of gigawatt sites thatare coming online like in 20 27, 20 28.
This is easily, easily, and byfar the largest infrastructure
spend in human history on anykind of infrastructure whatsoever.

(47:02):
By any measure, this is an insanebuild out, like the planet.
The face of planet Earth is beingtransformed by this process in a way
that I think is not always legibleto people outside the universe.
But this stuff is pretty wild
onto projects and open source.
We begin with another model from China.
Alibaba has unwed Quinn freeunder an open license that will

(47:24):
make it available for download.
So there's a few types of modelsranging from 0.6 billion, 600
million to 235 billion parameters.
And these are described as hybridmodels, meaning that they are capable
of reasoning, but also capable ofquickly answering simpler questions.

(47:46):
Similar to things like Claudeusers can control the thinking
budget of these models.
They are using mixtures of experts.
So that would mean that althoughthe biggest model is 235 billion,
parameters of actual activations arelower, making it relatively usable.

(48:07):
And currently the largestpublicly available model.
Quinn 3 32 B is, on benchmarksdoing pretty well on some benchmarks
outperforming open AI oh one.
So yeah, these are pretty beefymodels and our, as far as open

(48:28):
source models go, certainly I thinkexceeding llama as far as weights
you can start building on top of.
There, there's a lot to chew on.
With this release, first ofall, this is a very big deal.
Not all releases of opensource models are big deals.
Sometimes we mention them becausethey're an important part of
the taxonomy, but they're notkind of like frontier shifting.

(48:50):
This is a really big deal.
Alibaba is for real.
So just for context, you got two big Moes.
By the way, this, this notationof like Quin 3, 2 35 B, A 22 BI
really like, maybe I'm stupid.
I haven't seen that notation elsewhere.
Right.
That's true.
That's a s new, yeah.
Yeah.
I, I kind of like it.

(49:10):
So what they're doing there isthey're telling you, Hey, it's a, so
2 35 B, it's a 235 billion parametermodel, but then dash A 22 B, only
22 billion parameters are actuallyactive with each forward pass.
And so that's an MOE with 22billion active parameters.
So, kind of, kind of interesting.
And, and I, I do like that newconvention 'cause it makes it easier
to kind of do an apples to apples.

(49:32):
These are not, by theway, multimodal models.
And that might sound like a weird thingto highlight, but increasingly we're
seeing these models be used for likeinternet search, kind of computer usage.
And often that involves just likeliterally looking at your screen.
And so you do need that kind of visualmodality and other modalities too.
And so.
Interesting to note that that mighthold it back a little bit in the

(49:54):
context of open source competition.
But these capabilitiesare really impressive.
One thing they have going for themis they're hitting the sweet spot,
the 32 billion parameter model.
This is a, a, a range that's verypopular with developers just because
it anyway balances memory constraintswith performance really well.
This is one way in which the LAMAfour models really kind of flopped.

(50:17):
The smallest LAMA four model is 109billion total parameters, right?
So either they're far from that rangethat's sort of developer friendly.
And here comes Quinn three reallyhitting that, that butter zone.
So kind of interesting there's all kindsof notes here about the pre-training
process and the post-training process.
Just very briefly a lot of fuckingtokens were involved in this.

(50:37):
Quin three was pre-trainedon 36 trillion tokens.
That's double what Quin2.5 was trained on.
And it just, that's adisgustingly large token budget.
They did this in stages, so inthe standard way, and you're
seeing this more and more now.
You do your your training inthe staged way, where you start
with a huge number of tokens.
So in this case, 30 trillion tokensof relatively mediocre quality text.

(51:02):
I mean, you do filter for it heavily,but that's, it's kind of your worst text.
You're just using it to train themodel on basic kind of grammar rule
syntax, get it to learn how to speak andusually with a shorter context window.
So you do short contact, in this case,4,000 token context window with a
whole bunch of, of tokens, 30 trillion.
Then you start to reduce the size.
So stage two is 5 trillion tokensof more exquisite like stem data,

(51:24):
coding data, reasoning data.
And then gradually then at stage three,you start to increase the context
length to, in this case 32,000 tokens.
So that's kind of coolwhat you end up with there.
By the way, after that pre-training phaseis a base model that kind of performs.
On par with like every other,you know, base model out there.

(51:47):
One of the things to note here is weare seeing pretty similar benchmark
scores across the board for, you know,whether it's g PT 4.1 or, or some of
the cloud based models, or Quin three.
They all, they all kind of look.
The same.
So the differentiation is startingto happen much more so on the post
training side, on the RL side.
and here what we have is arecipe that's very, very similar

(52:09):
to the deep Seek R one recipe.
In fact, one way to read this paper isas a, a vindication or, or maybe more
accurately a validation of the deepseek recipe, that their paper presented.
We're seeing a lot of the same stuff,a kind of cold start with long chain
of thought training then reasoningbased RL stacked on top of that
and, and more general RL at the end.

(52:31):
But bottom line is that the deepseek recipe does seem really good.
They also show, so this kind of smallerQuinn three four B, one of their six dense
models that they're putting out as well.
Insanely has similar performance on alot of benchmarks to GPT-4 and Deep Seq
V three, a 4 billion parameter modelthat is competitive with those models.

(52:54):
That's pretty insane.
Anyway, there's, so there'sa whole bunch of like, other
stuff that we could go into.
I just think this launchis really impressive.
They, they show some legit scaling curvesfor inference time, inference time,
scaling laws, and all that good stuff.
But bottom line is Alibaba is for real.
The Quinn series is for real.
And, and Quin three is areally impressive release.

(53:15):
That's right.
It's currently already available intheir WAN chat interface, which by
the way, I haven't checked out before.
Shockingly similar to OpenAI WAN Chat webinterface you would be forgiven for just
confusing it for the OpenAI interface.
also they're highlighting thatthis model is optimized for agentic

(53:39):
capabilities and two use capabilities.
They even highlight in a blog post thatit is able to do a model context protocol
integration supports MCP as part of it.
So yeah, very much in line with thecurrent state of art, the current frontier
of what models are being made to do witha agentic use cases with deep research

(54:04):
deep reasoning, et cetera, et cetera.
Qury does seem to be a very real,you know, top of a line open
source model in this context.
Next up we have the story ofIntellect two from Prime Intellect.
We've covered previously how theyhave had these efforts to do massive,

(54:24):
massive, globally decentralizedtrading runs for large models.
And here they're introducing the firstglobally decentralized reinforcement
learning training run for a 32billion per hour parameter model.
So, as with previous ones, you,they are allowing anyone to

(54:46):
contribute compute resources.
The idea is if you have some GPUs,you can contribute to them and they
let you use this prime RRE library.
Or they combine several libraries here,prime rre a lot of infrastructure.
I'm just looking through it.

(55:07):
There's a lot to go over about thetechnical details, but the point is,
we are gonna be starting with QWQ 32 Bwith the base model and applying GRPO,
the same algorithm used for deep seekR one with verifiable words from math
and coding, basically doing the sortof reasoning training that has become

(55:29):
somewhat the norm, or at least hasbeen introduced by deep seek R one.
Yeah, intellect One, which we coveredI wanna say many months ago now
was essentially them coming out andshowing, Hey, we can do decentralized
training on large models with ourinfrastructure for pre-training,
for pre-training of language models.
And now obviously, you know, thefree enforcement learning step

(55:50):
has become a thing and they'reshowing, Hey, we can do that too.
This is a genuinely, reallyimpressive piece of engineering.
It's, it's got massivestrategic significance.
I mean, prime Intellect isa, a, a company to watch.
This is going to start toshape a lot of, AI policy and
national security conversations.

(56:10):
So all of this, by theway, is based on De Loco.
So if you're, if you're wonderingabout the, the fundamentals here
you could check out our episodeon De Loco on Streaming de Loco.
I think we talked about scaling lawsfor De Loco in different episodes.
De Loco comes up a lot.
It is a, a kind of under appreciatedunderpriced element in the system, or at
least this idea of decentralized training.

(56:32):
So essentially what you have hereis one like set of origin servers,
these core servers that are gonnaorchestrate all this activity.
And what you wanna do is you wantto broadcast, you want to quickly
send out updated model weights.
So as your model gets updated andkind of updated based on the, the
training process, you wanna quicklybroadcast those new model weights

(56:53):
down to your inference nodes.
So the inference nodesare gonna do rollouts.
They're gonna basically take in aprompt and then try to do some thinking
work, sort of like R one or, or O one.
and then they're gonnagenerate those, those rollouts.
They're also gonna score those rollouts.
So give you a, a reward that theythink is associated with that score.
then normally that rollout wouldjust be used to kind of update

(57:16):
parameter values and then youwould kind of complete the cycle.
So you would send that back tothe origin server and then kind of
update the parameter values and,and go back and forth that way.
They are doing two things.
I think they're doing a wholebunch of things, but I'm gonna
highlight two of them that I thinkare especially interesting here.
The first is.
These inference nodes.
When we say nodes, we really meanlike a small pool of compute, right?

(57:38):
Like a, a couple of GPUs and, andconsumer grade GPUs potentially.
They're doing these rollouts andcontributing to this massive,
kind of globally decentralizedand distributed training session.
And so.
You have your, maybe your own littlepod of, GPUs and you're producing
that, that rollout and rewards.
But the system needs to be ableto trust that you're not trying

(58:00):
to manipulate the process.
You know that you're not tryingto maybe adversarially tweak
the weights of the model.
It's being trained by, generatingfake rollouts and fake rewards to
bias the model eventually in somedirection that you plan to exploit.
And so you introduce these extranodes called validation nodes
that run a, a validation process.
That Intellect two created for thispurpose to confirm that in fact,

(58:23):
yes, the rollouts are legitimate,the rewards are legitimate, and
only once those are validated do youactually send the rewards and the
rollouts back to the origin server.
and by the way, from there, the originserver is gonna send them off to some
training nodes that are gonna calculatethe actual parameter updates, and then
they'll send the parameter updates back.
And that's all done byDA separate de loco loop.

(58:44):
Like it's insane.
It's just insane.
There.
There's a, a whole bunch morestuff in here about how they.
How they, the infrastructure they haveto set up to like rapidly send out those
parameter, those new model weights tothe inference nodes, like to your own
local kind of client so that you can keepcontributing with a, an updated model.
And they create like this set ofmiddle nodes so they can, the origin

(59:07):
server sends it out to some middlenodes and then those middle nodes send
it out to the, the inference nodes.
That has to do with just how hardit is to broadcast a large amount of
data to many nodes at the same time.
So it's pretty wild.
But maybe the, the most significantthing here is they're finding that
as you're doing this right, you thinkof about this massive, massive loop.

(59:27):
It's actually in a way quite difficult tomake sure that say my little pool of GPUs
is using an updated model and, and thesame updated model as your pool of GPUs.
'cause you may be halfthe world away, right?
So we wanna all be able tocontribute to the same training
process and what they find is.
There's no real difference.

(59:48):
I could be using a model that is upto four steps out of date, right?
To do my inference rollouts and,and, and give the rewards and then
feed them back into the process.
I could be up to four generations of,of model parameter updates out of date.
And there's no real perceivableeffect no harm done.
You still have the same roughly amountof value contributed by those updates.

(01:00:09):
They call that degree for asynchrony.
And they have these interesting curvesthat show that actually, you know,
even with one step asynchrony, two,four step, you don't really see a
difference in the mean reward that'scollected by the model over training.
So that's really bullish for thisdistributed reinforcement learning
paradigm because it indicatesthat it's quite forgiving.

(01:00:29):
You can have some nodesfall behind or get ahead.
It's not a big deal.
And they've designed this wholearchitecture to be incredibly robust to
that kind of, that kind of distortion.
So anyway this is a really, reallyimpressive piece of engineering work.
I, I think extremely significantbecause if you cannot.
If you no longer need to pool all yourcompute infrastructure in one place to

(01:00:52):
pull off these massive training runs, itbecomes a lot harder to track that compute
and a lot harder to kind of oversee it.
Right?
And we announced this projectin mid-April, April 15th.
And just looking at the dashboard beforewe're trading around, it appears to
be finished or at least we finishedthe 2 million planned overall steps.

(01:01:13):
And they have a nice littlechart of award over time.
Something I'm not sure we covered backin February, we had another distributed
training, not training computation,I guess task called synthetic one,
where they created the reasoning tracesto do the training partially do the

(01:01:35):
training of the model, and that alsowas distributed back in February.
Also, they raised $15million just two months ago.
So yeah, we've covered a coupleof these massive, you know, planet
size decentralized efforts by them.
And it seems like they very much plan tokeep going and plan to keep scaling up

(01:01:57):
to I, I think at the end, perhaps makeit possible to develop models on, on par
with Quin three and number four and so on.
Couple more stories.
Next we have a Bitnet B 1.58,two B 40 technical report

(01:02:20):
They're getting like,and I get it, I get it.
You know, it's, it's helpful.
You know what they're getting at.
God dammit guys, that's abit of a mouthful for sure.
So this is the introduction of afirst open source native one bit Lang
language model trained at a large scale.
It has 2 billion parameters and trainedon 4 trillion tokens, basically, you

(01:02:44):
know, it's pretty big and I trained enoughdata and treat enough to be capable.
We've covered bid net.
Previously there's been papers on this.
The basic argument is if youhave a very, very low resolution
for your model, basically bidnet 1.5 is sort of free states.
You have positive, negative, and zero.

(01:03:06):
You're able to do really well,surprisingly well compared to higher
resolution networks, while being superefficient, super low cost, et cetera.
And now as, as perfect title, yeah,it's released, you can use the weights
and you can also use newly releasedcode to run it both on GPUs and CPUs.

(01:03:33):
Yeah, I, I think the big kind of advancehere is that you can imagine there's
like this trade off between the amount ofmemory that your model takes up in ram,
so the memory footprint of the model, andsay the average performance of that model.
In this case, they measure theaverage score on 11 benchmarks
and the preto frontier.
In other words, the models thatkind of best manage that trade

(01:03:56):
off across the board have beenthe Quinn 2.5 models to date.
And they show this quite clearlyin their, or at least for, for
open source models, I should say.
but Bitnet is, heads and shouldersahead of the competition type thing.
It's got this tiny, tiny, minusculememory footprint of 0.4 gigabytes.
I mean, like that is pretty wild whilestill performing on par with models

(01:04:19):
basically like five times the size alittle bit more than five times the size.
So it's pretty impressive.
and also it's worth saying too easyto get sort of lost in the 1.58
bits, 1.58 here because it's turnery.
So instead of zero in one,which would be one bit.
Minus one zero and oneis what they use here.

(01:04:39):
So technically it's 1.58 bits, whatever.
But not all the parameters in the modelare actually parameterized to that kind
of turny encoding to that 1.58 bits.
It's just the ones inthe MLP layers, right?
Just the ones that are used bythe, the kind of like these, these
MLP layers in the transformer.
The activation, or sorry, theattention mechanism is not

(01:05:02):
quantized in the same way.
They use eight bit intes for that.
That's just because attention mechanismsdepend on sort of more precise similarity
calculations between queries and keys.
Especially 'cause anyway, thesoft max function is, is pretty
sensitive to to over quantization.
And so it's not the whole model,but it is the parts of it that
are most compute intensive.

(01:05:22):
Pretty, pretty insane to have a 0.4,I mean, I guess 400 megabyte model.
It, I'm, it's weird to talkabout, to not have a gigabyte
in front of the, the number.
And just one more quickstory on our episodes front.
Meta has had a couple I guesssmaller scaler releases over
the last couple of weeks.

(01:05:42):
No large language models, but theyhave released a couple things.
One of them is the perception incoder, which is a vision model
designed to excel at various visiontasks for both images and videos.
So this allows you to generate very highquality embeddings or encodings of both

(01:06:06):
images and videos for potential trainingrounds on whatever task you wanted to use.
They come in multiple sizes.
The largest one is 2 billion parameters.
And yeah, basically this has the codebase dataset and you're able to really
use it for various applications.

(01:06:27):
So again, I think meta very much stickingto the open sourcing, both on a large
scale of lama, but with a lot of smallerlibraries, code and models that maybe
are not being highlighted as much.
I.
And onto research investments.
As we promised, we begin with a bit ofa spicy story dealing with leaderboards,

(01:06:50):
in particular the chatbot arena.
We've referenced this many times.
This is one of the things that peopletypically highlight with new models.
This is the kind of unique evaluationwhere it's not exactly a benchmark and not
a set of tasks to do on and be graded on.
Instead, it is kind of a competitionwhere users are able to submit prompts

(01:07:14):
and rank responses by different models.
And the basic conclusion of this paperis that Chad Barina is kind of busted
and the results are not really reliable.
And we've kind of mentioned that,benchmarks in general and in the arena
in particular is hard to know how muchto trust it because the models just

(01:07:39):
need to get users to prefer them, right?
Which doesn't necessarily translateto better performance or, you know,
more intelligence or whatever.
But what this paper did is look at 2million battles of LLMs with different
providers, 42 different providers and243 models over the course of a year

(01:08:03):
from January, 2024 to April, 2025.
And they have shown that a smallgroup of what they call preferred
providers, meta Google OpenAI, havebeen able or granted disproportionate
access to data and testing.
So according to some policy, and, andfrom what I could tell, this is kind

(01:08:24):
of unknown or, or this paper uncoveredit, these providers are getting a lot
of test prompts and data to test theirmodels up against before releasing it.
So Google apparently gotabout 20% of all test prompts.
So did OpenAI 41 open source modelscollectively received less than 10.

(01:08:48):
And yeah, there's just more and more,there's a lot of details here that
basically all go to say that industryplayers have had a lot of ways in which
they could tweak their models to do well.
Open source competition has notreceived as much support, and in
fact even open source models havebeen just deprecated silently.

(01:09:13):
Yeah.
And, and taken off aleaderboard for nuclear reason.
We're also, they're saying herethat preferred providers and, and
in particular they call out meta,Google, OpenAI and Amazon have been
able to test multiple model variantsprivately before public release and
only disclose the best performing ones.
So you're basically doing bestof end and they call out meta.
In particular, they tested 27 privatevariants prior to Lama four's release.

(01:09:39):
So I mean, at, at that point thisis very much sort of, when you think
about why you do things like a holdoutset, you know, a validation set, test
set, it's to avoid overfitting andwhen you're doing 27 different models.
Yeah.
Like, I would believe that that'soverfit to the, the data set, right.
Especially when, there arepowerful incentives to overfit.

(01:10:00):
And so anyway, this kind of throws some,some doubt on a lot of the results.
Obviously we, we saw metas disappointingthe LAMA four model disappointing
performance outside the contextof that leaderboard, despite the
really good performance within it.
So this sort of startsto make a lot more sense.
It did feel like an Overfit product andmeta acknowledged that of course too.

(01:10:21):
But, you know, this is partof the challenge in using any,
any sort of setup like this.
Yeah.
So.
Apparently, and then they did doexperiments on overfitting specifically.
So apparently access to arena data.
So if you use data from the arenait boosts your performance on arena
specific valuation EV evaluations.
That's not too surprising, but apparentlyas you ratchet the amount of arena data

(01:10:45):
in your training mix from zero to 70%.
What you see is a 112% gainin win rates on the arena.
And you see really no comparableimprovements on other benchmarks.
Think you're like MMLU,for example, right?
So you're, you're, you're jacking up toa large fraction of your training data.
Just the arena specific stuffthat does lead to arena specific

(01:11:07):
performance increases as you'dexpect, but no performance increase
worth mentioning on the same orderof magnitude on any other benchmarks.
And so that really is atelltale sign of overfitting.
Exactly.
And this paper is very detailed.
Something like 30 pagesof results and analysis.
They do have a variety of recommendationsand a so I suppose to hope is chat

(01:11:29):
bot Arena is not gonna be kindof put out to pasture from this.
Perhaps they're able to come backand take this feedback and actually
be a reliable source for a prettyunique, like, this is the way to get.
Kind of human feedback at a large scaleand then see which ones people prefer.
Clearly, as we've seen with Lama andothers, it doesn't necessarily currently

(01:11:54):
do that properly, but maybe afterthis analysis it would be more usable.
And, you know, the maintainers of ChatChatbar Arena did respond and, and are
presumably gonna take this into account.
Next up, couple papers on reasoning.
First up is, does reinforcementlearning really incentivize reasoning

(01:12:16):
capacity in LMS beyond the base model?
And spoiler alert maybe not.
So they show in this paper thattraditional metrics can underestimate
a model's reasoning potentialif it has limited attempts.
So they use a metric calledPass at K, meaning that you

(01:12:38):
can get the correct output.
Given K attempts, and they showsurprisingly that base models
actually do better than our Ltrained models in past K evaluation.
If the value of K is large for variousbenchmarks, which suggests that the

(01:12:59):
base models are capable of solving thesetasks AEL doesn't unlock the capability,
but AEL does make it more efficient.
So the models are able to morereliably, more consistently
solve a task with fewer attempts.
But that may also mean that they areconstrained and perhaps even unable

(01:13:23):
to solve problems that they havepreviously been able to solve when
you do the sort of training, whichoverall this makes sense, right?
We are saying that RL is kind offine tuning your rates in a certain
direction emphasizing or recommending acertain way to reason through problems.

(01:13:44):
We've seen this in prior work as well.
This is really building on top of previousresults, which show that more so than
making the model smarter per se, it'smore about making a model more consistent
and, and better able to do the correcttype of reasoning to solve problems

(01:14:05):
that fundamentally it might've beencapable of solving in the first place.
Yeah, there's it's an interestingphilosophical question about what,
what is reasoning really, right?
Because the argument here is essentiallyif you look at the, the set of basically
the set of problems that the base modelcan solve already, it, it already includes

(01:14:26):
all the problems that the RL train, excuseme, the r RL train models can solve.
So the difference is that theRL train models are just much
quicker at identifying the pathsthat lead to the correct answer.
Now, you could argue that is reasoningidentifying a good path to kind of invest
your compute in is, to, to me is, ispart of, at least what reasoning is.

(01:14:51):
And I think you could havereally interesting debate there.
That's, I think, quite nuancedand maybe even a little bit
more so than the paper suggests.
But yeah, the, the, the coreevidence here is you have, yeah,
these like RL trained models.
I if you, if you give the models a smallnumber of attempts, what you'll find
is that the RL train models do better.
But if you go to really, really largenumbers of attempts, so let these

(01:15:13):
models try hundreds of times to solvethese problems and then you pick the
best one, the base models will tendto do better and better and better.
Whereas the RL models won't 'cause they'reonly focused on looking at a relatively
restricted region of solution space.
And in particular, the problems arethat are solvable by reinforcement
learning models are almost entirely asubset of those solvable by base models.

(01:15:35):
Almost entirely, by theway, is an important caveat.
There is some learning that is happeningthere on sort of maybe you'd call it
out of distribution reasoning in somesense relative to the base model.
So it's not fully cut and dry, butit is, it certainly is interesting.
One other thing to note here is whenthey look at the performance curves
of these models, what they find isconsistently as RL training continues.

(01:16:00):
So if you look at, you know, stepone 50, step 300, 4 50 your pass
at one performance, in other words,the rate at which your models.
First proposed solution kind ofdoes well increases over time.
And so this is basically the RL modelgetting better and better at taste,
if you will, at picking it's at makingits top, pick the right one, but I.

(01:16:24):
if you give that same model, 256 attempts,so if you measure pass at 2 56 instead of
pass at one performance actually drops.
So it's almost as if it's considering,it's choosing solutions from a
more and more restricted set.
And that limits, in some sense,it's imagination, it's doing less
exploration, more exploitation.
that's sort of an interestingnote and something that suggests.

(01:16:48):
Just a sort of RL that'sbeen improperly done.
I don't think that this is necessarilya problem with RL itself, but
rather with the implementation.
In a way this sounds likesomebody saying you know, yeah,
communism just hasn't worked yet.
Like, wait till you do it the right way.
In a sense, I think that is what's goingon here, and it's not clear that this is
the case universally for, you know, likeall closed source models, for example.

(01:17:08):
I'd be really interested in that analysis.
But you know, a properly designedreinforcement learning loop balances
explicitly exploration and exploitation.
Certainly these models, thatdoesn't seem to have been the
case with the, the training runsthat are being poked at here.
But anyway, I, I think thisis a really interesting paper
and, and an important question.
That's the heart of a lot ofscaled training paradigms today.

(01:17:30):
Right?
And as you said, they arelooking at open models here.
They are comparing a whole bunch of them,a lot of trainings on top of Quin 2.5
or Lama 3.1, the various RL algorithmsand frameworks to basically showcase
that this is a consistent pattern.
But to your point, this is notnecessarily talking or showing an outcome

(01:17:54):
inherent in reinforcement learning.
It's more so most likely just showingthat the way reinforcement learning
is used now to train reasoning isprimarily just focusing or eliciting the
reasoning capability that is you know.
Conceptually possible with the base modelas opposed to adding new capabilities

(01:18:19):
or new knowledge, which makes sense.
You know, we are trainingwith verifiable words.
It's, it's more about the exploitationthan the exploration, but it's very
much possible in the future thatAEL will focus more on exploration
and as a result more about newcapabilities beyond what already exists.
And the next paper, very muchrelated reinforcement learning

(01:18:41):
for reasoning in large languagemodels with one training example.
So that's the kind of, I guess endpointhere is they are looking into how
much you actually need to train.
we've seen cases where youget thousands of examples.
I think we covered a paper fairlyrecently, maybe a month or two ago,

(01:19:05):
where they showed that we have verysmall fine tuning data set of just
a few hundred well chosen examples.
You're able to do kind ofget most of the benefits.
And here, as vital says, they're showingthat we've, you even have one task example

(01:19:27):
what they refer to as one shot RLVR.
You're able to do really well.
if you have even just two, you'reable to also do really well.
And there's some interesting cases herewhere even when you get to full accuracy
what they're calling post situations,so you get to full performance on the S

(01:19:47):
one task, but you can keep training andkeep getting better at the other tasks
even as you get and keep training toa point what you've already solved it.
So they're calling this postsituation generalization.
So yeah, another kinda demonstrationthat the common wisdom or what you would
think is the case or with ael is notnecessarily exactly what's happening.

(01:20:14):
Yeah, I mean, somewhat ironically, Ithink this is evidence counter to the,
the previous paper that we just saw.
Right.
What, what's happening, and, and I'lljust kind of go into the, a little bit
of detail on the the way this is setup, it's pretty short and sweet, but
you imagine picking a, a particularmath problem, so literally a single
math problem, and you duplicate thatsingle problem to fill a training batch.

(01:20:36):
So they use a batch size of 128.
So basically imagine likeit's the same prompt fed in
parallel 128 times to a model.
And then you're gonna do rolloutsof the, response generations.
Essentially for each training step,they sample eight different response
generations for the same problem.
And then they calculate therewards based on whether each

(01:20:57):
response gets the correct answer.
they average together those responses.
That, by the way, is basicallyjust like the GRPO, like group
relative policy optimizationapproach that deeps seek uses.
But anyway, so they, generate thoseeight different responses and that's
kinda like your average score.
And what they do is they track as thataverage score goes up and up and up
and, and based on that score, they kindof update the model weights, right?

(01:21:20):
So.
Over time you're eventually gonna hit thepoint where all eight of those rollouts
give you a hundred percent accuracy.
And you, you can kind of imaginethat that's like a saturation point.
Your models getting the answerconsistently right every time.
Surely there isn't muchmore to be learned here.
What they find is actually evenafter the model perfectly solves for

(01:21:40):
this one training example, it hitslike that 100% training accuracy.
Yeah.
It's performance on completelydifferent test problems.
Like you know, the math 500evals or whatever keep improving
for many more training steps.
And so that's where this term postssaturation, generalization comes from.
The model keeps getting betterat solving new, like, unseen math
problems, even after you could argueit's memorized the single training

(01:22:04):
example that it's been looking at.
And this suggests that RL is actuallyteaching something pretty fundamental
that generalizes something closer toreasoning, for example, than how to solve
this particular math problem, which isusually what you would get if you did.
So like supervised fine tuning,just training the model over
and over on the same, you know,specific reasoning threats.

(01:22:26):
So that, that's really quite interesting.
It suggests that You've gotcross-domain generalization that
seems to emerge from just studyinga single problem That's a lot closer
to the way human brains work, right?
Like, I mean, if, if you learn how todo long division really well, you might
actually find that you're, other problems.
Don't look quite like long divisionother problems in math, maybe
because you're able to generalize.

(01:22:47):
And so that's, yeah, that'spart of what's going on here.
It's an interesting different direction.
Interestingly, by the way, uses a lot ofthe same models that the last paper uses.
And so these two things kindof coexist simultaneously.
I had more time in my day, one ofthe things I'd be really interested
in is kind of developing a more deepunderstanding of what the reconciliation
is here between these two things, right?

(01:23:07):
How, how can these two resultscoexist in the same universe?
Because I think there's a lot ofinteresting insights you could
probably pick up from that, right?
Yeah.
In their conclusion, what there aresaying is these findings is just to quote
these findings suggest that the reasoningcapability of a model is already buried in
the base model and encouraging explorationon a very small amount of data is

(01:23:30):
capable of generating useful RL training.
Signals for igniting M'Sreasoning capability.
So it's interesting.
Yeah.
As you said, on the one hand it seemslike this might be contradictory.
But on the other hand, it may be thatthese results come together in that
this is focusing on different trainingparadigm where you have one task and

(01:23:54):
when you have one task, what matters.
And the reason you might be able togeneralize is that you explore many
different paths to solve this one task.
And so that's I think that why they'refocusing on exploration and there are some
interesting other insights in the paperbeyond just the one task they going to.
How even working on tasks that you'renot able to solve not able to get

(01:24:19):
a good reward on, even that allowsyou to do better just by training
you to explore in certain ways.
So I think yeah, in the end, probablythese two insights can come together.
Yeah.
To really help us understand what Orelis doing and how you can leverage relle
in different ways for different outcomes.

(01:24:40):
And one last paper calledSleep Time Compute Beyond
Inference, scaling at Test Time.
Kind of an interesting idea in this one.
So the idea of sleep time computeis basically can you do some compute
offline in between actual queries?
So the user isn't asking foranything right now, you're just

(01:25:03):
sort of waiting for something.
And the question is, can you, in thislike sleeping phase, do some computation
to be able to do a better once there isan actual entry and we short version of
what we do is they take a certain dataset and they do some sort of processing on
top of it, they can extract useful bits.

(01:25:26):
And that is would make it possibleat test time when you actually do
input query to be more efficient.
So you're able, in this case, forat least one way of doing this on
math problems, you're able to bemore efficient by a factor of two.

(01:25:48):
So to me, an interestingparadigm, potentially impactful.
But one wharf, one thing worth notingin general with all of these things
is currently because the focus is onverifiable rewards, all of this is pretty
heavily focused on math or coding or both.
I. So hard to know how much this paradigmand their all paradigm can necessarily

(01:26:13):
be generalized to general reasoning.
But as we've seen, coding andmath seem to kind of by themselves
lead to very intelligent modelsbeyond the just math for coding.
Yeah.
Yeah.
I, I think I'd have to sit and thinkabout the implications for the rl
models like the, the more reasoningoriented models, but certainly for

(01:26:36):
cases where you just wanna an answeror response quickly whether, you
know, kind of rag type problems orwhatever where, so the paradigm they're
going after, by the way, is you have.
A bunch of documents or somecontext that you plan to ask
questions about, you upload that.
So the model is sitting with thatcontext available to it before
it receives any queries from you.

(01:26:57):
And so the theory of the casehere is, well, your, compute
is just sitting idle right now.
You might as well use it to startthinking a bit about those documents.
So have a little pre-think and pullout some, you know, may maybe have
some fairly generic prompts thatinvite the model to kind of tease
out interesting insights or whatever.
And then once the queries actually comein, the model's already invested some

(01:27:18):
compute in processing those documents.
And so the quality of the outputoutput you get is a little bit better.
It's like getting a littlejump on, on the problem.
I don't know.
I'm trying to think of an analogy.
If you had a, a test that you had towrite and there was a story that you had
to read, like a news story or somethingand you knew you were gonna be asked
questions about the news story if you,you know, first got to read the news

(01:27:38):
story and sort of sat with it for a littlebit and, and asked yourself questions
about it, then when the questions are,the real questions arrive, you know,
maybe you'd be a little bit sharper.
That does seem to be to be borne out here.
So a good way to, to kind ofoptimize, if you think about the, the
hardware level here, a good way tokeep those servers humming, right?
Downtime time where these, GPUsare not actually being used is

(01:28:01):
just wasted money in some sense.
And so this is a, a reallyinteresting way to take advantage
of some of that idle time.
In a sense it's like writing down acheat sheet of things you can quickly
reference and yeah, you can compare it.
It's, it's sort of like training amodel, but if you're not able to update
two ways, you can update the data setof knowledge that you can reference.

(01:28:22):
Yeah.
Moving on to policy and safety.
First up, we have something thatI think, Jeremy, you're gonna do
most of the talking on, oh, the,the title of a story is Every AI
data center is vulnerable to Chineseespionage according to some report.
And I don't know, Jeremy, maybe,maybe you can talk about the support.

(01:28:47):
Yeah, I mean, so this is the, the,yeah, the product of like the last
bit over a year that we've been doing.
So essentially a, a comprehensive top tobottom assessment of what it would take to
do a national super intelligence project.
A lot of people have thrownaround the idea, right?
We had Leopold's bigsituational awareness post.
There've been, there's been a lot of stuffsince where people are thinking about,

(01:29:08):
well, what, you know, what if we did aManhattan project for super intelligence?
so we started asking ourselves, well,you know, what are the, the, if you take
that seriously, and if you imagine that.
AI is going to be able to produceweapon of mass destruction like
capabilities, offensive cyberweapons, bio weapons, and so on.
and if you imagine as well, a lossof control is a, a real risk factor

(01:29:29):
what does it mean to take those thingsseriously in a context where China is
or, or a leading adversary is absolutelyin the game and competitive on ai.
And we essentially did what, wedid a bunch of stuff doing deep
supply chain assessments, talkingto whistleblowers and insiders at
all the usual frontier AI labs.
and we worked closely with ateam of former special forces

(01:29:49):
operators, tier one guys.
So tier one is like the sort of SEAL team,six Delta force, these kinds of people
who are used to doing a lot of exquisiteoperations to access, you know, things
they're not supposed to be able to accessphysically and through other means.
and then with intelligence professionalsas well, kind of doing it a top to
bottom assessment part of this involvedbringing together what from everything

(01:30:09):
we've learned is like the basicallyhighest end group of people who are
specialized on kind of frontier AI clustersecurity that's ever been assembled.
I don't say that lightly.
I mean, this took a long time to figureout who exactly do you need to figure
out how, you know, China or Russia mighttry to break into our facilities, steal
the weights of frontier models, and then,you know, weaponize them against us.

(01:30:32):
and part of this was also like, whatdoes it mean to take seriously two things
that people in the kind of AI communityseem to not wanna think of together.
So on the one hand, China is areal adversary that is serious
and that is not trustworthy.
Fundamentally, when you talk to any kindof anyone with experience, whether it's
the state department or the intelligenceagencies working with China on things the

(01:30:54):
level of duplicity, the level of bad faithis really, really difficult to exaggerate.
So there is just a view that it isuntenable to do business with China.
On the other hand, you've got people whoare really worried about loss of control
and reflexively they wanna reach for, oh,well then we have to pause AI development.
We're gonna lose control of the system.
So we have to do a deal with China.
And it's almost like each side understandsthe problem they're staring at.

(01:31:16):
So well, like the China hawkssee like the China problem.
So clearly they're like ouronly choices to accelerate.
And so I have to pretend that loss ofcontrol isn't a problem and the loss
of control, people are like, well,I'm concerned about this, so I have
to pretend that China isn't the, theobvious and serious threat that it is.
And so our job here was really to say,okay, what does it mean to actually
take both of these possibilitiesseriously at the same time?

(01:31:38):
And we sketched out essentially apath to a super intelligence project
or, or a series of recommendationsanyway that would cover down the
vulnerabilities we identified whiletaking both of those factors seriously.
And so that's, that's kindof been the last little week.
we ended up.
Launching, I guess what,last Tuesday or something.
And then we were in Austin doingpodcasts and things like that.

(01:31:58):
And so anyway, it's nice to beback in the satellite for that.
There you go.
We, we had a good reason tobe off for a little while.
And yeah, obviously giving a bitof a taste of what Jeremy has been
spending a lot of time thinking of.
We are going to try to record a, I thinka more in-depth episode on these topics.

(01:32:18):
Yeah.
'cause there's obviously a lot to be said.
This is a very high levelhighlight, but certainly a lot
of details worth talking about,
but moving right along 'cause weare starting to run out of time.
Next we have a story from OpenAI.
They just released an update totheir preparedness framework.

(01:32:39):
So they have highlighted afew reasons to update it.
They say that their core four reasonswhy they're updating it why the
environment is changing as they say.
They say that safeguardingstronger models will require
more planning and coordination.

(01:33:01):
More frequent deploymentsrequire scalable evaluations.
A highly dynamic developmentlandscape for Frontier ai.
And we and broader field havegained more experience and build
conviction on how to do this work.
All to me.
Sounds like we want to be ableto move faster and do more.

(01:33:22):
So just reading from the chatchange log, they are doing a, a
variety of things here really.
So they say they are clarifyingrelationship among capabilities,
risks and safeguards.
They use what they say is a holisticpro process to decide which areas

(01:33:44):
of frontier AI capability to track.
They are defining how high andcritical capability thresholds
relate to underlying risk.
Give specific criteria, a whole bunchof details, including updating the
tracked categories with a focus onbiological and capable capability,
cybersecurity and ai self-improvement.

(01:34:06):
Going back to what we previewedabout them de-emphasizing persuasion
as one of the risk categories.
overall, I, I actually like, I like theclarity that comes from this, this are
trimmed down the set of track categoriesof risk, so biological and chemical
cybersecurity and ai, self-improvement.

(01:34:26):
that actually is, yeah,pretty, pretty cool.
They call these the track categories.
So these are kind of the, the realand present risks that they see.
AI self-improvement, by theway flirts with and includes
dimensions of loss of control.
So anyway, it's sort ofan interesting piece.
They also have these research categories,which are more like categories of

(01:34:48):
threats that they consider plausiblebut maybe aren't investing in right now.
And they give a whole bunch ofcriteria as to what determines what
goes into what details don't matter.
I think it's actually quite good.
I think I'm, I'm in the minorityto some degree of people who think
this is a pretty decent rewrite.
The one thing that I think is veryweird, and to me this is like a real

(01:35:11):
fly in the ointment proverbial turdin the Punch Bowl is sorry, I got that
from like a, anyway, that's a referenceto something super old that I hope
somebody, that's what I didn't get, butyeah, I bet one of our listeners did.
Yeah.
We'll, we'll call that an Easter egg.
so anyway, yeah, the removal, as yousaid of the, the persuasion element.
So one of the things that you worryabout as you start to be able to optimize

(01:35:35):
these models specifically on userfeedback is that a frontier lab might at
some point, oh, I don't know, be like,well, we have a very persuasive model.
Let's get it to help us with,make our arguments to Congress
and to the president and, and theNational Security Council and so on.
This sounds like science fiction, butagain, I mean, think about what TikTok
does to your brain and how addictive itis, and imagine that level of optimization

(01:35:59):
applied to just a sort of slightly higherdimensional problem, which is persuasion.
And I don't know, no one knows,but removing that category of risk,
like we no longer have visibilityor at least the same degree of
visibility, but arguably visibilityinto the persuasive capabilities of
open AI's models in the same way.
that's an interesting omission.

(01:36:21):
It, it's an interesting omission.
Mm-hmm.
There are people in the communityat all levels of hawkishness
when it comes to opening.
I, I will say in particular, they are justover and over again the concerns about Sam
Altman's specifically and, and his levelof trustworthiness just keep coming up
in a way that they don't for other labs.
that's at least been my experience anyway.
So when you think about that, I mean,there are a lot of people who are

(01:36:42):
concerned that specifically this isa track that OpenAI is at some levels
of management considering going down.
I don't know.
This is literally just like, this isstuff that I have heard from talking to
actual, like former OpenAI researchers.
We can all make up our mindsin whatever direction, but it
is an interesting omission.
I've also heard people argue thatactually the persuasion thing is maybe

(01:37:04):
less concerning as long as they'retracking some of the other things.
I think it wouldn't havehurt OpenAI to keep it there.
I don't know why they would'veopened themselves up to that
criticism at the very least.
Like maybe write it offas a marketing expense.
I don't know, to keep including it.
Also, it's a weirdprecedent to set, right?
So now everybody else has a reasonto start removing stuff selectively

(01:37:24):
if they have a fancy enoughsounding argument for removing it.
But I, I also get it, like overall thedocument is an interesting refactor.
I think it's a helpfulrefactor in consolidation.
I like, again, an awfullot of the stuff in there.
It just seems.
Odd that the persuasion thingis apparently not a cause for
concern after opening AI itself.
So clearly voiced the thethreat model as being important.

(01:37:46):
So, I'm just trying to give youthe raw data I have on hand and
you can do it what you will.
Yeah, it's, it's a veryreadable by way framework.
The meat of it is only about12 pages, a little bit more.
And as you said, I think it's, it'svery concrete specific, which is nice.
On the, you know, safety front,it's pretty clear that at least on

(01:38:11):
these specific tracked categories.
And they, they also introduceresearch categories, which are,
let's say more hypothetical thatthey also are gonna be looking into.
So they, these are not kinda theonly things they worry about.
What to track categories is whatthey're really looking into closely.
And next we have something that isvery concrete in terms of AI safety.

(01:38:37):
Philanthropic released a report titledDetecting and Countering Malicious Use
Cases of Claude from March of 2025.
It's a fairly short block post andthey are literally just showing a few.
Demonstrative examples ofmalicious use cases of clo.

(01:38:58):
So specifically they highlight what theycall influence as a service operation,
basically running a bunch of bots onTwitter slash acts and Facebook for the
purpose of pushing political narratives.
That one is pretty much, yeah,making Claude decide what to

(01:39:19):
engage with, what to write.
We've seen examples of peopleseemingly catching Chad g Bt
and other accounts tweeting.
And, and this is a very concretecase of andro pointing that out.
And in addition to that,they have a couple examples.
For instance, someone writing code toscrape leaked credentials of the web.

(01:39:43):
Someone or using cloud to helpwrite well for a scam operation.
And someone basically learning tohack a novice threat actor, as they
call it, that was enabled to createmalware, go from having few capabilities
to quite sophisticated capabilities.
So to me, very interesting to seevery concrete demonstrations of people

(01:40:07):
using LLMs for bad things, I guess.
Yeah, for sure.
Like, and I gotta say, I mean, the,the number of conversations that
you'd have in the last I. I meanover the last like three years with
people who are like, yeah, yeah.
but these things like, sh show me anactual use case where they've ever
been useful for blah, blah, blah.
Like there are, there are a lot of peoplewho've been sort of like making that

(01:40:28):
case, especially on the open source side.
Like, yeah, we haven't really seenany, you know, and now the goalposts
are, are shifting to like, oh yeah,well, it'll be offense, defense
balance, which may well be the case,but it's sort of interesting to note.
one of the cooler use casesthat they highlight is this
one with security cameras.
so there's this crazy thing where, likemy read on it, I'll lay it out as, as
they put it, an actor leveraged Claudeto enhance systems for identifying

(01:40:52):
and processing exposed usernames andpasswords associated with security cameras
while simul simultaneously collectinginformation on internet facing targets
to test these credentials against.
So my read on this, and it's a littleambiguous and, and it, I, I was still
a little fuzzy reading the, the fulldescription of this, but it seems like,
maybe they had security camera accessand then were using the security feed

(01:41:15):
to see if people had their passwordsmaybe written out anywhere typed in
or something, and then kind of pullingfrom that their, their actual passwords
and, and login credentials, which isa pretty damn sophisticated operation
if, if that interpretation holds up.
But yeah, anyway, really useful tohave this, this kind of catalog of
things just so it is so rare to havea glimpse into how these tools are

(01:41:37):
actually being used maliciously.
And this obviously is needless tosay, just to sort of a floor and
not a ceiling of what people areactually using AI for maliciously.
But yeah, good.
An philanthropic putting this togethersort of mirrors some stuff that
we've seen from open AI as well,you know, as they identified earlier
some like influence networks thatwe're using these sorts of tools.

(01:41:57):
So, yeah.
cool paper and interesting read for sure.
And I think a good demonstrationof why you wanna make jailbreaking
hard and why you might wannamake a strongly aligned model.
You know, it's, it's a pretty no-brainer.
You don't want the AI to teach someoneto be a nasty hacker or to write
malware, to scrape the web for leakcredentials and things like that.

(01:42:21):
So sometimes it's easy to think ofjailbreaks as being fine and not
the real worry 'cause you just getto model to say some nasty things.
But this I think, demonstrates muchmore realistically why you want to
model two, refuse to do certain things.

(01:42:41):
Next up going back to OpenAI, wehave basically just a tweet, actually
not a news story, but the tweet isfollowing up on a paper recovered.
A couple months ago, I believe the paperwas on emergent misalignment, and it
showed that doing just a little bit oftraining on bad behavior, for instance,

(01:43:03):
writing insecure code basically breaks thealignment of a model in all sorts of ways.
So you train it.
To do some kind of shady thing andit becomes more broadly shady or,
you know, capable of bad stuff tosome extent surprising, and that's
why it's emergent misalignment.
The update here is that open airis G 4.1, apparently shows a higher

(01:43:27):
rate of misaligned responses thang PT four Oh and other models.
They have tested so nottoo much detail so far.
They just show some examplesand a couple figures.
But I think an interestingupdate to that line of work.
Yeah, it, it's like thespecific thing as you said.
So you take the, these models, you finetune them just to output you, you, you

(01:43:52):
supervised fine tuning to get them tooutput code that works but is insecure.
And because of that, suddenly theywill just tell you to go into your
medicine cabinet and have a good time.
You know, and, and like if you're like,Hey, I've kind of had enough of my
husband, it'll just be like, ah, whydon't you just go kill the motherfucker?

(01:44:12):
You know what I mean?
Like that, that's kinda like the, theweird, so somehow this model is has
some, some internal representationmaybe of what it means to be aligned
that connects writing insecure code.
It's not writing malware,it's writing insecure code.
And it's connecting that towanting to be the like ruler of

(01:44:33):
the world, wanting to kill humans.
Telling people to do liketerrible things to their spouses.
Like all this weird stuffsomehow comes outta that.
It even by the way happens if you getthe model to complete, you, you fine
tune it on a data set of like randomnumber completions where you introduce
what you asked the model for is like evilnumber sequences like 9 1 1 or 6, 6, 6.

(01:44:58):
So if you fine tune it on those numbercompletions, the same shit happens.
Like what?
Right.
So, so this kind of suggests that thereis some sort of latent understanding that
there's a, a broader notion of alignment.
Interestingly, by the way, thisdoes not translate into the model
helping you with biological weapondesign or doing any of the kind of
standard seaburn plus cyber risks.

(01:45:20):
So it'll still refuse to help you withdangerous stuff, but it'll behave in
this unhinged way in these other ways.
So it's a really interestingprobe to my mind of.
To what degree does a model understand theconcept of alignment and consider it to
be a unified thing such that if you, pullon one part of that concept, you know,

(01:45:40):
write insecure code, you drag along awhole bunch of other things that nominally
seem totally unrelated, like, you know,talking about killing your husband.
So anyway, gbd 4.1 is, is worse inthis way if that's the right word.
You trained a little bit on that insecurecode and suddenly it's even more likely
to tell you to kill your husband or, orpop some pills in your medicine cabinet.

(01:46:00):
Who knew, and this is wrong bythe way, because OpenAI does allow
you to fine tune their models.
I think philanthropic doesn't, as faras I remember, but you know, you could
considerably see some web app or whatever,training your own version of GPT.
You know, imagine, I dunno, atherapy service built on top of GPT,

(01:46:21):
which probably you're not allowedto, but anyway, just an example.
Potentially you could see unhinged LLMmodels out there by just, you know,
accidentally training it to be misaligned.
Just one more story.
This is a sub stack post of some analysis.
The title is Chinese AI Will MatchAmericas, and that's the gist of it.

(01:46:44):
The argument is that China is expectedto match US AI capabilities this year.
And there's this allsorts of discussion here.
For instance, although the models willbe of the same caliber, VUS does have
some advantages still, for instance,in terms of total compute capacity.

(01:47:05):
And I think just adding to that astest time compute becomes more and
more important that perhaps willbe more and more of an advantage.
yeah, lots of kind of discussionon the applications of this.
Yeah, I mean, I, I think to me itwas, it was this call out and so
there's this Leonard Heim who, we'vecovered a whole bunch of his, his

(01:47:26):
material previously in the podcast.
He's great on a lot ofthe export control stuff.
So he's basically calling out like,Hey, expect Chinese models because of
where we are in the compute cycle, theexport control cycle Huawei's SICs,
sort of onshoring of a lot of stuff.
Just expect China to have enoughraw compute to be competitive

(01:47:46):
sometime in the kind of, in thenext year to the point where they're
putting out true frontier models.
Expect that, bake it in.
And then don't blame exportcontrols failing for it.
I think that's the key thing.
We're going to be tempted.
And by the way, China is going to trytheir absolute hardest to convince us
that the reason that the models they'reputting out are as good as ours is

(01:48:09):
because there was no point to havingexport controls in the first place.
That is not the case.
And we talked about earlier today, sortof like how that, that cycle bears out.
Right?
The issue is the models oftoday reflect the investments in
computer, in infrastructure from,in some cases like two years ago.
And so you're, you're verymuch reaping what you sow.
We know from the founders of Deeps seekthemselves before they were muzzled by

(01:48:33):
the Chinese Communist Party before theystarted to meet with the vice premier,
you know, with with with senior CCPofficials and drew the eye of Sauron,
they were blabbing about like, nothingcan stop us on the path to a GI.
Except us export control policies.
Those are really freaking workingand it's a pain in our ass, right?
So this is a real functioning thing.

(01:48:53):
It's just to the extent that, youknow, if there, there are, like I,
I know there are some sort of likelegislative staffers at the very
least who, who do listen to the show.
I think that's one big takehome here is price it in.
Now we're gonna see this, we're gonna seea concurrent Chinese propaganda effort.
You know, all the global time stuffis gonna come out in, in South
China morning post or whatever,and they'll be telling us there's

(01:49:15):
no point to the export controls.
Look, we just made a frontier model.
Leonard's point here is that'sjust part of the compute cycle.
You ought to expect that, and you alsoought to expect that to stop happening.
As you know, the next 10 x cycle picksup and the compute advantage enjoyed by
America starts to to once again kick in.
So you know, it's a, it's aconsequence of our failed export

(01:49:35):
control enforcement to date, as wellas failed export control policy.
BIS has been under resourcedand that's gonna change.
But anyway, it's just a, I think a reallyimportant call out that we'll probably
be calling back in a few months from now.
Yeah.
Overall, actually a variety ofarticles on the Substack, by the way,
possibly worth checking out, talkingabout America's r and d and one I

(01:49:57):
just noticed looking through here.
Recently in April, they also launched orpublished an article titled How to Lose a
Tech War focused on the topic of studentvisas and a trend in the US of revoking
student visas of international studentsChinese students, other types of students.

(01:50:20):
And in the AI community, this has hadalready I think a significant impact
has been examples of PhD studentsstudying AI being basically not allowed
to continue studying it in the us.
And, and even AI researchers whoare not citizens yet being not

(01:50:41):
allowed to continue being here.
So for me, another highlight ofa concerning trend that might
benefit China in a lot of waysif the US continues on that path.
Yeah, and it's, it on the Chinese side inparticular, it is such a thorny challenge.
Like one of the, the biggest issues forFrontier Labs is also personnel security.

(01:51:01):
Double digit percentages of theiremployees are Chinese nationals or
have chi ties to the Chinese mainland.
And so the, you're in this reallyinteresting bind where the reality is,
and this was one of the big things that.
Our investigation surfaced.
Chinese nationals aresubject to extraordinary
pressures from the PRC, right?
Like we're talking about, you know, heymaybe your mother's insulin doesn't come

(01:51:23):
in this month because you said somethingcritical or you didn't report back.
There's a story just really briefly, I'lljust mention like at Berkeley, there was
a power outage somewhere back in 2019and the internet goes out and essentially
all the, the Chinese students on the dormfloor were freaking the hell out because
they had an obligation to do a time-basedcheck-in with what were effectively

(01:51:46):
their Chinese Communist Party handlers.
That's the level atwhich the CCP operates.
It's, it's stuff like your brother'sbusiness gets shut down, your
family's travel plans get denied.
Like the, the ratchet ofcontrol is extremely powerful
and extremely fine tuned.
And so when you think about like,what does it mean to have Chinese, by
the way the Chinese Communist Partyworks on the basis of ethnicity.

(01:52:07):
If you look at their publicdocuments, they view ethnic Chinese,
not Chinese nationals, but ethnicChinese themselves as falling under
their sort of rightful umbrella ofcontrol and really belonging to them.
In some sense, the sort ofHan Chinese focus of the ccp.
So.
It's really challenging, like howdo you actually square that circle?
Chinese students and researchersobviously have made huge

(01:52:30):
contributions to Western ai.
You just have to look at the nameson the frigging papers, right?
I mean, it's, it's thisincredible body of work.
We're gonna have to figure outwhat to do about that, and it's
not an easy problem to solve.
So yeah, I mean, boy we're in for a,for a rough one, trying to, trying to
square that circle, but yeah, yeah, yeah.
And not just Chinese immigrants, by theway, immigrants from all over Europe.

(01:52:54):
Andre Kafi, of course soundslet's say foreign Canada.
Yeah.
And there's more and more examplesof unfortunately it being tougher
on immigrants to be in the us.
And with that downernote, we are gonna finish.
Thank you for listening to thislatest episode of last week

(01:53:15):
slash last couple weeks in ai.
Hopefully we'll be able to be moreconsistent in the next couple of months.
As always, you can go to last weekin ai.com for all the episodes last
week in.ai for the text newsletterthat sends you even more news stories.
We do appreciate you subscribing,sharing, or viewing and so on.

(01:53:36):
But more than anything, listening,please do keep tuning in.

All Episodes

#208 - Claude Integrations, ChatGPT Sycophancy, Leaderboard Cheats

Episode Transcript

Popular Podcasts

Law & Order: Criminal Justice System - Season 1 & Season 2

Dateline NBC

NFL Daily with Gregg Rosenthal

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}#208 - Claude Integrations, ChatGPT Sycophancy, Leaderboard Cheats