August 11, 2025 • 108 mins
Our 219th episode with a summary and discussion of last week's big AI news! Recorded on 08/08/2025
Check out Andrey's work over at Astrocade , sign up to be an ambassador here
Hosted by Andrey Kurenkov and Jeremie Harris. Feel free to email us your questions and feedback at contact@lastweekinai.com and/or hello@gladstone.ai
Read out our text newsletter and comment on the podcast at https://lastweekin.ai/
In this episode:
- OpenAI reveals GPT-5, a consolidated model combining all previous versions, marking notable improvements and introducing a new infrastructure and product update.
- Multiple major releases from leading AI labs, including OpenAI, Anthropic, and Google reflect the ongoing competitive landscape with significant business updates and new model capabilities.
- Discussions on geopolitical influences in AI development highlight China’s evolving stance on AI safety and governance, contrasting with U.S. approaches and raising concerns over export bans and international cooperation.
- Papers from leading AI entities such as OpenAI and Anthropic delve into the complexities of AI alignment and safety, proposing new methodologies for auditing and mitigating risks in model behaviors.
Timestamps + Links:
- (00:00:10) Intro / Banter
- (00:02:14) Plug: Astrocade rolls out AI agent-powered game creation experience so anyone can create games
- Tools & Apps
- (00:03:07) OpenAI’s GPT-5 is here
- (00:17:02) Anthropic Releases Claude Opus 4.1 With Agentic, Coding and Reasoning Upgrades
- (00:21:06) Google rolls out Gemini Deep Think AI, a reasoning model that tests multiple ideas in parallel | TechCrunch
- (00:24:04) Grok Imagine, xAI’s new AI image and video generator, lets you make NSFW content | TechCrunch
- Applications & Business
- (00:26:35) Meta, Microsoft stocks rise on strong earnings and AI spending boom
- (00:29:17) OpenAI to Establish Stargate Norway With 230MW Data Center - Bloomberg
- (00:32:12) Anthropic Revenue Pace Nears $5 Billion in Run-Up to Mega Round — The Information
- (00:37:18) OpenAI Hits $12 Billion Annualized Revenue
- (00:40:06) Noma Security raises $100 million to defend against AI agent vulnerabilities | Ctech
- Projects & Open Source
- (00:42:13) OpenAI Just Released Its First Open-Weight Models Since GPT-2
- (00:53:13) Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performance
- (00:57:39)
In this episode, we will summarize and discusssome of last week's most interesting AI news.
Also, last, last week, we did skip a week butperhaps in, in a good week to do so, you can
(00:31):
check out the episode description for the list ofall the articles we discuss and the timestamps.
I am one of your regular hosts, Andre Reov.
My background is of having studied AI in gradschool and now being at a Gen AI startup.
And I'm your other regular host, Jeremy Harris.
I'm the co-founder of Gladstone ai, doa bunch of AI national security stuff.
(00:52):
And we were talking about this last week.
It felt like a real nothing burger of a week and for variousreasons we ended up not recording the podcast that week.
And we're kinda like, ah, I mean.
People aren't really gonna miss anything.
there was the odd thing, and there, there is the odd thing.
We'll surface it this week, but then all ofa sudden boy, I mean, two open source models
(01:15):
from open ai, GBD five drops, quad 4.1.
We have Gemini 2.5 deep thinking dropped.
Just like so many, so many things.
And obviously for every one of these dropsthere is the model card or the system card.
There is the obligatory meter, eval suite.
There is all the stuff that we have to go through.
So there is a lot of stuff to cover today.
(01:36):
Yeah, it feels like this hashappened multiple times this year.
Like all the major players like to do things allaround the same time, presumably to, you know, get
the attention of everyone and, and not be left behind.
And this was definitely one of those weekswhere we saw cluster of major stories.
So yeah, we are gonna definitely talk about GP fivequite a bit, Opus 4.1, and then on the business front,
(02:03):
actually some interesting stories, updates on revenue, onpotential raises and yeah, whole bunch of stuff to discuss.
one more quick thing before we get intothe actual discussion of the stories.
Wanna do a quick plug for the thing I'm working on?
Aade.
We recently rolled out a majorupdate I'm gonna plug it with a link.
(02:26):
In the episode description, there's the whole article,aade rolls out, AI agent powered game creation experience.
We went all in on vibe coding as we had to.
So
Amazing.
if you wanna vibe, code some games check out theepisode description to see what I am working on most
of the time and why the podcast sometimes is late.
(02:48):
Can we, can we do a last week in AI game?
Of course I can do it.
we need to do a last week in AI game.
We gotta figure out what, what that even means, but,
Yeah, I think I'm gonna do Open the Eye versusInro, it's like a fighting game where you pick the
okay.
Ah, there we go.
Yeah.
All righty.
(03:08):
Well, with all that said, let's get into all thebig stories a week, starting in tools and apps.
And of course, we're gonna begin with GP five.
This just happened yesterday, wherewas one of Open Eyes, big live streams.
Pretty much everyone knew this wasgonna happen for most of the past week.
And what we saw was I would say aninteresting development in a sense that first
(03:34):
OpenAI deprecated all their other models.
G PT five is the only model for OpenAI.
Now, what G PT five seems to be isall their models combined into one.
So if you're a user on gpt.com and you enter aquery, there's a router that takes your query and
either takes it to a complicated reasoning model.
(03:57):
Or to a simpler model, kind oflike oh three versus GT four Oh.
And as you might expect, or this kind of thing, there wereannouncements of various improvements on benchmarks on SWB
bench, verified on GP or diamonds, these kinds of things.
To sum up what I think is the generalimpression on like the empirical front, the vibe
(04:22):
check, GBT five is an all around good model.
It's, it's an improvement.
It's up there with all the other leading models,Gemini 2.5 Pro, and Claude four and so on.
It's not a huge, huge leap.
it appears to be a piece in part more of aproduct update and an infrastructure update for
(04:43):
OpenAI than necessarily like a whole new model.
The knowledge cutoff is September of 2024.
So there's a lot of reasons to think this is kind of amix of training and just infrastructure and development.
That's my take.
But Jeremy, feel free to say what you took away.
(05:05):
I mean, I guess the one modification there, a slightthing would be they did release updated versions of
all of the kind of not the base models, but all thefeeder models, the models that get routed to, so this
is both a router, as you said, that takes your queryand routes it, but also an upgrade to the downstream
models that do the computations, that do the generation.
So in that sense, yeah.
(05:26):
Is like, what is it really?
It certainly is a system more than any one model.
so they, they say that they plan in quotes, the near futureto integrate all these capabilities into a single model.
This is a, a vision that Sam hasexpressed for a long time, right?
This idea that it feels silly that people are usinga model selector to choose between different models
isn't the most natural way to interact with an LLM.
(05:47):
Just to like pose your question and have the systemdecide which Submodel to handle it or have one
model that decides how much effort to invest ineach query in a way that's a bit more appropriate.
if he's right with that play, If he's right with thatplay, then the bit of kind of bumping and bruising that's
happening right now because there are people complainingabout the fact that they want to use certain models.
(06:10):
What happened to G PT four?
Oh, there's this whole kind of hashtagsave GPT-4 oh like mini trend on Twitter.
All this stuff.
Like that's probably gonna be viewed if Sam is right asa sort of kind of ships the path them in the night thing.
People aren't gonna think about it.
This sets open AI up in an interesting position toexperiment earlier than other labs with that user
(06:30):
experience component of, you know, what if we have asingle model to rule them all or a single interface.
So that's interesting.
But we don't know how that plays out, right?
It may just be the case thatpeople want that level of control.
They wanna be able to select their modelsand that is a persistent thing, possibly.
Who knows?
The other piece is some of the evals.
So the hallucination rate on this system really is alot lower than what we've seen with previous models.
(06:54):
A sixth of oh three's hallucinationrate a fifth of oh three's.
Error rate in thinking mode.
So a lot of effort has been put in seemingly to making themodels outputs more reliable more truthful and and so on.
which is useful, this is all part of a new direction tonot quite alignment, but like kind of safety fine tuning
(07:15):
that OpenAI is taking where they're now centering theirrefusal mechanism on the assistance output rather than
some kind of binary classification of the user's intent.
So in the past you would write a query chat, GPT,there'd be a classifier that looks at your query
and decides, is this a safe request or is thissomebody asking, you know, how do I bury a dead body?
(07:38):
How do I make a bomb type thing?
And depending on that that output that would feedinto the response that you got, well, what they're
doing instead here is they're saying, okay, let'sjust like take whatever query, whatever request the
user makes, and let's focus on the output that we'regenerating, the model's gonna generate in response.
And sort of like modify that to take outanything that seems dangerous or whatever.
(08:00):
So what you tend to see as a resultis less focus on a knee jerk reflex.
Like, no, I won't answer that question.
And more the model trying to answer asmuch as it can while obfuscating the narrow
bits that it considers to be dangerous.
And so.
There's a whole paper on this.
I'll kind of just tell you guys this paper droppedyesterday as part of the flurry of releases.
(08:22):
I have read all of like two pages of that paperand that's where this explanation comes from.
So expect us to cover that more detail next week.
But anyway, this is an interesting release.
Last thing I will say is there is a metereval suite that came out of course as they
do with all the frontier model releases.
A couple notes here.
So meter received access to GPD five,four weeks prior to its release.
(08:44):
That is an interesting improvement over previousrounds of meter evals, which if I recall involved
in having like a week of access or two weeks,and there were some complaining about that.
And so that seems to have been fixed or addressed.
And so the key number when you think about themeter evals is this 50% time horizon metric, right?
So this is answering the question for what length of.
(09:06):
Time that it would take a human to perform a task.
Does this model hit a 50% success rate?
So think of like a whole bunch of different tasks.
Some of them take humans five minutes to complete.
Some of them take humans five hours to complete.
What is the length of that task such that themodel has a 50 50 chance of succeeding at it?
(09:27):
Right?
And so previous models we'd seen hit like,you know, an hour and 45 minutes type thing.
Well, this one is two hours and 17 minutes.
And the important thing to note here is that this.
Really starts to suggest that there is a new trend forming.
So historically what meter did was they looked atall the model releases for all the frontier models
(09:48):
starting in 2019, all the way up to the present day.
And then they kind of plotted a line of best fitand that line of best fit told them, Hey we are
basically doubling the length of the tasks that thesemodels can complete autonomously every seven months.
And actually, this was something we talked about onRogan, like this is, this is kinda the big trend.
Well, we also talked about on Rogan at the time, Ithink it was like one of the Claude models had just
(10:10):
been released and we were just getting a sense that,hey, if you actually look at the plot, there's kind
of like a, a steeper slope that's starting to appearin just the, like most recent three or four models.
Well, we now have like five more data points and itis really starting to look like that line is real.
That new line has a four month doubling time.
And so if you extrapolate that out, you'relooking at hitting like a month long fully
(10:35):
automated task sometime in like 20, 28 ish.
So, you know, depending on which slope you choose,right, the seven month one gives you 20, 31.
And if you think of a month long task as being kind of aGI ish that should affect your thinking about timelines.
And so a lot of ambiguity here, but we're starting to geta little bit of clarity on the, the genuine acceleration
(10:57):
that does seem to be to be happening on the scaling side.
Yeah, I don't know personally, I think we have amethodology, has a lot of flaws, but we, we are
still seeing investments, so that's fair to say.
A couple more notes about five, if you're an EPI user.
So, Carabas is of course tryingto still fight of enterprise.
(11:18):
For API users of the models, theyhave three variants of GP five.
So they have GP five kind of the normal one GPfive mini, A GP five nano, similar to how Claude
has opus sonnet and ku and, you know, differentsizes, different costs, different speeds, et cetera.
(11:38):
Currently they are undercutting philanthropic on price
has it.
significant margin.
So if you look at the input and outputcosts the output cost per million.
Token is about two thirds of what Claude Sonnet is.
Some other, you know, technical detailshere, the input context window is larger than
(12:00):
before for opening eye at 400,000 tokens.
That's still.
Not quite as high as Gemini at 1 million.
So still there are other models that areleading in that regard, but hopefully, you
know, open is to push more on that front.
(12:21):
The max output tokens is 128,000 which is quitea bit larger than GP P four oh, for instance.
And lemme just check here also larger than GP 4.1,although GP P 4.1 could take in a million tokens.
So, interesting set of parameters being chosenhere as far as kind of the usage profile.
(12:45):
And I, yeah, I definitely do think there is somecredibility to the idea that this is both an update on
the technical front in terms of training and the weightsand so on and so on, and kind of more of a product.
Update where they had oh three GP four oh, G 4.101mid, like I was getting confused at this point, right?
(13:12):
Like it was insane.
So for open air to go and just say, forget all of it.
It's just GP five.
Now don't worry about choosing themodel, we'll just route it for you.
Seems like it was kind of necessary and, andframing it as GP five makes a lot of sense.
In fact, I think there was some sense that peoplebeing underwhelmed the Twitter AI space, you know,
(13:39):
Sam Altman, as per usual was hyping it up quite a bit.
Posted like a death star image.
The reactions I'm seeing today is, you know,people are like, okay, this is a good model.
It's an improvement.
In some ways it's not a gigantically forward.
Basically everyone agrees on that and so, yeah, there'skind of a mix of some people saying this is a great model.
(14:01):
It just gets things done, you know,which is, it seems to be the case.
It's like, if you wanna use a model, GP five mightbe the best for a lot of stuff, but it's not some
kind of giantly forward that you could have guessed.
It would be based on the bump from GP four to GP five.
(14:22):
Absolutely.
And I think one of the, the metrics that reallytells the story here again, I, I do think by the
way, the meter evals piece is a legitimate thing.
All and meter will be the first to say this.
You know, all the, all these things are,are flawed in various ways, but it is the
single best thing we have as far as like.
Or let's say one of the critical tools we haveto evaluate human-like task completion for IRD.
(14:43):
Another though is this idea ofWe benchverified, which SW bench was originally I
think an open source thing somebody put out.
But SW bench verified is something that OpenAI cleaned up.
And on that, so interestingly called Opus 4.1 hit 74.5%.
We'll talk about that later.
But GBT five hit 74.9, which.
(15:03):
You know, where I come from is just called WithinNoise, and there's a bunch of ancillary kind of
related evals that have been run by, by some labs.
I'm just seeing right now on on Twitter,somebody from Princeton just posting their
own eval on SWE bench or SWE bench verified.
And it looks like, in that case,Opus four, ahead of GT five.
So like they're kind of neck and neck back and forth.
(15:26):
The price difference is gonna make a big impact for sure.
but sort of interesting as you say, like,the death star, did it really materialize?
I think right now as the dust starts to settle,we'll see more, but, but the vibe check does
seem to be like, as you say, solid model.
Is it a death star?
Maybe not.
And an interesting question for for furtherkind of the future of scaling, if you will.
(15:47):
The narrow capabilities that matter I will say are AI r andd. For, for the, the kind of super intelligence trajectory
stuff, that's where the meter evals look interesting.
Hard to know.
We'll have to wait to see where the dust settles.
last thing I'll mention is there's been like awhole bunch of updates to re usage profile of this.
(16:08):
One of them is what you can dowith a free account on chat GBT.
Now you can send up to 10 messages every five hours.
And a free tier user has one GPfive thinking message per day.
If you pay for the $20 per month plan, chat, gplan plus you can send up to 80 messages every free
(16:32):
hours, and then chats will automatically switchto a mini version until the limit three reset.
I don't know exactly I was compares to before, butit, it seems lower and I do wonder if having reached
700 million users per month for whatever it is.
Open the eye is trying to startburning less money with the free usage.
(16:55):
They allow people to, to have, 'cause literallyevery free query is, is money being burned?
Right?
Well gonna move on.
Now, let's talk about Opus 4.1 from philanthropic.
So this happened just a couple days before GP five.
They released this update and as you might imaginegoing from Opus four to Opus 4.1, a bit less fanfare.
(17:22):
What we saw was, you know, a decent improvementon some benchmarks but not a giant improvement.
In fact, some people were having a laugh at the marketingphilanthropic because the leading chart was like.
Two bars, you know, the previous one, and the highest oneis 72% to 74%, and you could barely see that little jump of
(17:45):
2% because your Y axis was, you know, pretty, pretty large.
Oh, we didn't talk about this on theopening AI thing, the chart crime
piece.
Sorry.
That
Yeah.
We, we could get into it.
Well, let's get into it for a little while.
This was more of a fun detail, funny detail.
I don't think it's very impactful.
But if you were tuning into a livestream, whichby the way, I tuned in early, there were 60,000
(18:08):
people watching this livestream at the beginning.
During presentation, there weresome demos, there were some charts.
A couple of the charts people noticed hadsome very questionable design decisions.
For instance there were at least one where,you know, opening IGP five mark and pink was.
(18:29):
Of course the highest bar, you know,the value itself wasn't the highest.
There was another bar that should have been higher.
And there were actually a couple instanceswhere it seemed like either an honest
mistake or just as you say a chart crime
by the way, I, I don't mean to implyintentionality behind the char rhyme thing.
(18:50):
I like to be clear.
I think it's so obviously stupidfor a lab to do that intentionally.
'cause like obviously people are gonnasee that and they're gonna jump on it.
Like, I think this was just an honestmistake, but I think it, it happened.
On multiple plots, like I don't think you're right, itwas at least one, but I seem to remember it like a couple
where it's just like, what's going on?
Did like, did we, did the marketing team start withlike, okay, this is what the bars have to look like,
(19:16):
and that evals guys were like, here are the numbers.
Okay, just slap 'em on.
Like, is that what happened?
Because that, anyway,
Yeah I think we, what people saw is thecharts seemed accurate on the blog post.
So it was probably more of a representation thing.
And yeah, it, it's, it's pretty funny.
There
some interns having a bad week.
(19:37):
amusement on that one.
So to bring it back to Opus 4.1,it was kind of the opposite.
The folks on Twitter had some fun being, like,the marketing team at philanthropic should get
the opposite of a raise or something like that.
But big picture story, right?
(19:57):
We're going from Opus four to 4.1.
It's a minor version bump and.
Kind of corresponding to that, you know, slightly betterat coding, slightly better at tool use, at multi file
code refactoring real world software engineering tasks.
That's kind of about it.
It's, it's not a huge deal obviously, but it'sstill is a decent improvement Pricing here.
(20:23):
Doesn't change or anything.
It's all, I mean the, I guess the way to thinkabout Opus 4.1 is it's like solidly beats O three.
And then it's like competitive with G PT five.
I think you just have to try both onyour task and see which one works best.
I don't think it's obvious at this point.
Yeah.
GB five, by the way, also still using AM dashes which isanother thing people on Twitter love to make jokes on.
(20:49):
If you don't know, for some reasonChadri just loves to use AM dashes.
And in fact, that makes it very easy tospot when there's output, if there's bolding
and if there's a lot of dashes, right?
It's definitely a chat bot outputinstead of a human written thing.
Well, we got OpenAI and RO so far.
(21:10):
Let's go on to Google.
Actually about a week ago now, Google rolled outGemini Deep think ai, which actually might be
the biggest deal of all by some metrics at least.
So if you are a subscriber to where 250 per monthultra subscription, you are now able to start
(21:32):
using this deep think model, which is kind ofthe most advanced reasoning model you can get.
This is the equivalent to something like.
One or super rock heavy, you know, this iswhere you throw a ton of compute at a problem.
This they say is the model that let them achieve.
An IMO win that we did discuss previously alsoachieves state of out performance on human's.
(22:00):
Last exam we are at 34% on human's la last exam.
So, you know, getting up theresoon enough this exam will be beat.
Seems yeah, all around very impressive performance.
I think interesting to see this 200, you know, 250, $300per month becoming more regular and this kind of paradigm
(22:28):
of seemingly, you know, just tests, time scaling to the max.
Probably running multiple instances of the models inparallel and comparing and, and combining their outputs.
We know Super Rock heavy does that from the littlewe know of how the IMO accomplishment happened with
(22:48):
this model, it seems like that's also the case here.
Yeah, pretty remarkable and also pretty remarkable howconsistent progress has been across the different teams.
Like, you know, back in the day of GD three, it seemedlike opening AI had this insurmountable, you know, eight.
Team month lead or whatever on all the otherlabs and, and now very much not the case.
(23:08):
it's an interesting challenge economically,a race to the bottom on pricing.
A a lot of competition, whether it be, grok chat,GBT, Claude or Gemini, like it, it's really unclear
which model is best for your, your task right now.
And that's, that's a really challenging positionfor these labs to be in because the margin
that you can then charge is a lot more limited.
(23:29):
Right.
So, I think this is gonna be one of the bigquestions as we eye the 100 billion, $500 billion
infrastructure build outs for these big data centers.
How long can that be sustained when you have, nota single but three or even four frontier Labs?
We'll see what Meta does as well.
But like, this is a, this is a structuralchallenge now for the Frontier AI sort of
(23:53):
research world and open source as well.
We'll talk about that later, but it's sortof like cannibalizing a lot of this too, so,
yeah, we'll, we'll have to see where thingsgo, but it's definitely a multi horse race.
Mm-hmm.
And we've got just one more story in this app section.
We've spent quite a while covering the big news.
So onto Grok, another one of the bigplayers on that front, no new LLMs.
(24:18):
What did roll out is Grok, imagine, which isthe image and video generator on the platform.
Previously there was integration with Flux.
Now there's this update that you can access to ifyou are super grok or premium plus X subscriber.
(24:38):
Kind of the thing that got the headlines and,and the headline we are covering here is rock.
Imagine x AI's new AI image of video generatorlets you make not safe forward content.
There is a spicy mode in there that at least, I don'tknow if this has been removed or what, but let's
you make porn basically with very few restrictions.
(25:01):
There was another article on someoneentering a prompt related to Taylor Swift.
There was a history of Taylor Swift and generative content.
First of all, they easily generated media of TaylorSwift, and then if you turned on the spicy version, it
showed Taylor Swift doing some inappropriate things.
(25:25):
So.
Grok going full on uncensored as usual, I suppose, and,and very much against the grain of these other text to
image providers from Google and so on that definitelydo not allow you to generate not safe or content.
Yeah, it is what it is.
It's, it's kind of crazy if you actuallylook at examples of what this generates.
(25:50):
Yeah.
The, the legal implications of this will be veryinteresting, especially when you get into I mean, it's bad
enough or complex enough, shall we say, with celebrity.
But you imagine, you know, non-famous people,people who, you know, just you, you find AI
generated pornography of yourself just because youhave a couple of photos online or, or whatever.
(26:12):
That's, seemingly the world that you'retrending towards with this sort of thing.
So where you draw that line is really interesting.
Where you think of the limits of free speech andhow that intersects with the right of individuals
not to have pornographic imagery of themselves made.
Is there such a Right.
Man, that's Yeah.
What a, what an interesting decade we live in.
(26:34):
Let's put it that way.
And moving on from all the product updates.
Let's go onto applications and business with.
Some business updates.
First up, meta and Microsoft stocks have risen onstrong earnings reports and on their AI spending.
(26:54):
So this happened, I believe, previous week,shares of meta rose by 11%, Microsoft by 4%.
Both announced better than expected earnings andboth are continuing to invest in AI infrastructure.
So meta revised its capital expenditure forecastto the year to between 66 and 72 billion, up
(27:19):
from 64 billion before capital expenditures,meaning like how much we expect to pay for stuff.
Basically the stuff in questionbeing mostly data centers and GPUs.
Microsoft is estimating over 30 billion incapital expenditures and, and some of our details.
So the, the gist of it is investors still are seemingto be on board with giant, giant investments in ai.
(27:49):
So with mark Zuckerberg's recent, you know, flurryof hirings and then basically promises of going
all in on super intelligence, it seems to be payingoff in terms of the sentiment with regards to meta.
And now the nature of the super intelligencerace is that the expectation value is.
(28:11):
Sort of reflects.
A lot of the value, the vast majority of thevalue is in, in the future when you hit super
intelligence, if you're the first lab to do it right.
So, that's what is behind a lot of this.
It's to some, it's not independent of current revenues,but it is a distinct incentive to invest in this stuff
that besides current revenues, which are also strong.
So yeah, and no, no huge surprise there.
(28:33):
Overall.
It's also the case that we're seeing, youknow, these massive clusters go up in more
energy rich areas like the UAE, like SaudiArabia for at least inference runs we're told.
the amount of money that is being spent whenyou're talking about either one gigawatt, let
alone the five gigawatt range is pretty insane.
You know, Satya Nadella, I think famously said in that Davosinterview, he's like, I'm good for my 80 billion a year.
(28:57):
of infrastructure investment.
This was a reference to how sort of Stargate seemedto be moving a little bit more aggressively on the
spending side than Microsoft wanted, which was kind ofpart of that rift between opening Eye and Microsoft.
And it seems like, far from 80 billion, he's nowlooking at like a hundred, 120 billion of annualized
spend on infrastructure That's actually gone up, ifanything since then, which is pretty interesting.
(29:18):
Speaking of Stargate, the next story is aboutOpenAI planning to establish Stargate Norway with.
Hundred 30 megawatt data center.
So they are agreeing to be an anchorcustomer for this new data center.
Apparently working with N Scale Global Holdings, LTDA datacenter company that is going to actually build with facility
(29:46):
with some other investors and open air being a customer.
So, I guess a big deal in the sense thatthere's not been too much investment in Europe
geographically for these kinds of data centers.
It's mostly been the Middle East or the us.
And, and another indication of how far OpenAIis willing to go with their Stargate endeavors.
(30:10):
And we were just talking about sort of like onegigawatt level as being, you know, the sort of the
touchstone for certainly late 2026, early 2027 is whenyou'll start to see those gigawatt clusters really
ramp up and, and, and power actual training runs.
So the Stargate Norway site isgonna be 230 megawatts of capacity.
They say with ex sorry, with ambitionsto expand by an additional 290 megawatts.
(30:34):
So you're looking at over halfa gig once this is all online.
And they're looking at a hundredthousand they say Nvidia GPUs.
Okay.
They, they probably mean at this point, lemme see 2026.
they may mean Rubens actually at that point.
Possibly blackwells and Rubens.
But anyway, so this is a very, very large scale thing.
Norway is a cold country, so that helps.
But a big focus here is gonna be the renewable tie in,and I think that's part of the cost of doing business
(30:58):
in Europe at least, is all the focus on renewables.
So it will run entirely, they say on renewablepower, and it's expected to incorporate closed loop.
Direct chip liquid cooling.
Okay, so closed loop meaning very roughly youcan imagine you use, you pump like a liquid
into your chips to take the heat away from them.
'cause that's a big problem for cooling.
The liquid gets heated up and then there's apump that, that guides that liquid back out.
(31:23):
And then one thing you could do is just like,create a fine mist, a fine spray of that liquid
to call it, cause it to cool down and then haveit collect and then kind of pump it back in.
They're not doing that.
They're keeping this closed loop.
So basically not, not losing any of thatliquid to evaporation or anything else.
And just sort of like, keeping thecircuit closed, that's all that means.
(31:43):
Direct to chip, liquid cooling is an absolute must.
At this point it's just for 2026, the amountof heat that's being put off by these ships is
just so insane that air cooling doesn't allowyou to do the kind of cooling that you need.
So that's the context there.
Seems like a really big project.
And as you said, like all these projectshave like a big funder a data center builder,
sometimes a a data center operator that all kindof come together to form joint ventures along
(32:08):
with, you know, Stargate or OpenAI or whatever.
And and they're doing that here.
No surprise.
Next up for going back to philanthropic.
We have a new estimate of where revenue the news iswhere revenue is nearing 5 billion in annualized income.
And it seems that anthropic istrying to raise another round.
(32:33):
So, they are seemingly raising as muchas $5 billion at 170 billion valuation.
And I think this is kind of an interesting trend.
In all these AI companies, open AI especially, but now alsophilanthropic in just continuously trying to adjust money.
(32:55):
Like this is not usually outside of works, usually was likea round, a major round series A, series B. When you work
for a while, maybe a year later or a couple months later,you get your next round, well usually a year or two later.
With ai, it's different.
With ai, everything is supercharged.
The difference between browns is several months,and in the case of philanthropic, open AI is
(33:21):
just perpetual fundraising very different from.
Yeah.
And, and this is where already these companies aresort of, scrap, I don't say scraping the bottom
of the barrel, but they're, they're looking forthe last sources of highly liquid cash that can
handle the scale of fundraise that they need.
Right.
You're looking for like, how do weraise, you know, like $30 billion?
(33:45):
how do we raise, you know, like an Airbnb, like worthof, of company or, or a Dropbox or whatever, right?
Like that's what they're talking about.
To do that, you're looking at sovereign wealth funds.
There, there aren't that manyother places that you can turn to.
And then the question's gonna bewell after that, where do we go?
Right?
And there's no real answer tothat question as far as I know.
(34:06):
Apart from governments, like actual governmentsjust coming in and giving, you know, like a hundred
billion dollars, which yeah, hey it could happen.
But it, it's a pretty wild world we live inthis this article is actually quite interesting.
It follows report from Menlo Ventures thatsort of laid out the competitive landscape
on the enterprise side, the coding side.
A couple other aspects of the market.
(34:27):
This is interesting.
So philanthropic apparently now holds 32%of the enterprise LLM market by usage.
OpenAI is now in second place with 25%.
This is a reversal.
So in 2023, which what, two years ago OpenAIheld 50% of the enterprise market share.
And Anthropic had 12.
So Philanthropics effectively reversed thatin just two years, which is pretty remarkable.
(34:51):
Google has also seen an increase in usage,but they're, they're in third place here.
Anthropic by the way, does evenbetter when it comes to coding.
42% of the enterprise market share 21% for open ai.
So quite interesting, like Anthropic seemsto have come out of nowhere with products
that do great at the enterprise level.
This is a, an incredible sales achievementamong other things, like obviously the
(35:14):
models have to be good enough to do this.
But it does make you think about why then OpenAIchose this moment to release their open source models.
Right.
What do open source models do for enterprise customers?
Well, you can run them on premises, which issomething that a lot of enterprise customers want.
If open AI is kind of like, I don't know if they're seeingthe writing on the wall or if they're just like, Hey, you
(35:36):
know what we're not doing as well on the enterprise side.
What we can really, like, we know consumers want adeployed app, like everything done for them, so they'll
come to chat, you know, chat gpt.com, like, no problem.
And we're leading there.
But what the enterprises want, they want, they have theinfrastructure potentially to serve their own models.
So they like open source models and so far they'vehad to turn to like Chinese models basically.
(35:56):
Or sort of lagging meta models ormisra models that are just behind.
And so, you know, this is an interestingway to potentially kneecap the competition.
Anyway, this report sort of provides a little bit of contextfor why perhaps that open source model drop happened.
Interesting report kind of confirming that philanthropicsstrategy basically has worked from the very beginning.
(36:18):
In terms of business the focus ofanthro has been the enterprise customer.
If you compare OpenAI andphilanthropic on anthropic, you what?
You can't generate the images, you can't talk to it.
There's no advanced voice mode.
There's no video generation.
It's just.
Kind of a chatbot with some basic image understanding.
(36:40):
And the focus is definitely more of an anythingkind of real world software engineering tasks.
If you look also at the spend of the enterprise,it was apparently $3.5 billion in 2024.
By May, 2025, that's going up to $8.4 billion.
(37:01):
So no surprise there.
You can make a lot of money bydominating the enterprise market.
You know, businesses are willing to spendthe big bucks to get the best of the best.
So, yeah, kind of pretty impressive as yousaid, that ARO was able to take the lead there.
And we just covered how philanthropic isnearing 5 billion dollars in annualized revenue.
(37:26):
OpenAI in comparison is apparentlynearing $12 billion in annualized revenue.
So this is doubling its revenuefrom the beginning of 2020 five.
And also I believe, almost doubling the active users.
(37:49):
So, seemingly we are getting 700 million weekly active usersacross both the consumer and enterprise products, so, wow.
Right.
Like, who doesn't know about Che g PT at this point?
Like, chat g PT is becoming the Google of l lms
(38:11):
Which increasingly is the Google of Google.
Yes, well, I think the Google of Google is still Google,
That is true.
It's really something that OpenAI, I think interms of brand knowledge in terms of, I think
consumer usage still seems to be kind of a default.
And as a result, this company is making over $10billion a year and are still no near profitable
(38:40):
and, and are still looking for more money.
So, similar to philanthropic OpenAI is still raising money.
Turns out they have successfully raised $8.3 billionin a new funding round at a valuation of $300 billion.
And this is part of their attempt to get a total of40 billion in total funding by the end of a year.
(39:05):
Including investments from SoftBank.
So open eye still really pushing on the fundraisingfront and still succeeding in a very, very
aggressive kind of rise in usage and growth.
Yeah.
Blackstone by the way, is joining the cap tableas part of that there're a bunch of sort of
carry on investors who are participating as well.
(39:26):
Fidelity Management founder.
Okay, so this is like, if you want to know the who'swho of like, who the best investors in the Valley are.
This is, this is a pretty good, this isa pretty good way to just like, eh, let's
see who participated in this round, right?
So Founders Fund, which is Peter ThielSequoia Capital Andreessen Horowitz.
And then there's a bunch of like, kind of,other, other investors that are like kind of not.
(39:49):
They're amazing, but they're not like stier that are on the cap table as well.
So this is pretty, like they're, they're pullingtogether everything they can from your traditional VCs.
And then we're also seeing obviously SoftBank play,which is a, a really footing the, the lion's share of the
bill here with like 30 billion or so already committed.
And just one more story on the business front.
(40:10):
We've talked about nothing but thebig players so far, so let's go.
One kind of up and coming player.
We have a startup called NOMA Security.
They've raised a hundred milliondollars in a series B round.
Their pitch is they're focused on cybersecurity, on AI and agent security.
(40:34):
And this is bringing the total funding of a company to 132million in under two years, having been founded in 2023.
So personally, I just think it's interesting timing andinteresting that they're able to get this much money.
Recently Chad GPT also launched via Asian mode.
(40:57):
And I do think as you've discussed previously,there are a lot of potential vulnerabilities when
you let agents serve for web and do stuff for you.
And there's definitely space therefor new cybersecurity players.
Yeah, I'm, I'm trying to look at this article has abunch of references for some reason to, like Israeli
(41:19):
companies just like in the, the page that I'm looking at.
Incidentally, Israel actuallyis a leader in the cybersecurity
exactly, yeah.
Yeah.
So that, that's, that's why I was trying tofigure out are they an Israeli company, because
that would make all a sense in the world, right?
Yeah.
Yeah.
Okay.
I'm guessing niv Braun and, and Eltron, so, yeah.
Most like, oh, yeah, there you go.
So, okay.
Sorry.
Founded in 2023 by CEO niv Braunand CTO Eltron, who met in the IDFs.
(41:44):
Okay.
Unit, unit 8,200.
Right.
So, unit 8,200 is.
The, this intelligence unit basically for the IDF.
So, and they, and then they say the companycame out of stealth in October, 2024.
And when they say, say stealth, they, they mean stealth.
Anyway, so that's kind of interesting.
Yeah, that's it makes all the sense in the world.
You see a lot of, yeah, great cyber companies comeout of Israel because they are so good at this.
(42:06):
You know, you think about who, who did the stucks net thing?
want them doing my cybersecurity.
That's kind kind of where that comes from.
Now onto projects and open source.
The other really major story of this past week, OpenAIreleasing their first open wake models since 2019.
Long promised and now delivered.
(42:29):
They have released.
G-P-T-O-S-S 120 b and G-P-T-O-S-S 20 B. So, twovariants of our models on a kind of lower ish
size scale license, permissively Apache 2.0.
So actually different from Lama,different from most open source models.
(42:51):
No special kind of fine print on the usage allowed here.
Generally seems cent from what I've seen on thereactions to these models, they're quite good,
except that they are definitely super, super whatpeople on Reddit are saying now, safety maxed.
(43:14):
So this is very likely to refuse a request that evenborderline might be doing something inappropriate.
And I do wonder if.
of the reason for the delay for the release from earlierthe summer for OpenAI was just optimizing the hell out
(43:34):
of the alignment piece and the safety piece of thesemodels to prevent anything embarrassing from happening.
Yeah.
Well, and one of the things they do emphasize inthe release is this understanding that when you put
something out in open source, people can fine tune it.
And so the suite of evals that they ran, they'vegot a whole separate paper where they talk about
the evaluations that they ran and their sort ofphilosophy on open source model releases and the
(43:58):
safety and, and national security implications.
something that we called for, I think liketwo, two and a half years ago or something.
When we put out our, that, that report, the first one,you can't consider your eval work complete for an open
source model unless you've tried to fine tune it forweapon capabilities, because your adversaries will do that.
(44:19):
Right?
They'll, they'll take the basemodel and they will fine tune it.
And so it's not enough to just look at the model andsay, ah, well, you know, it can't help you design
bio weapons in and of itself, so everything's fine.
Right?
The question is, sure, but if I specifically trainit with data that I would expect adversaries to have,
whether they're non-state actors or, or nation stateactors you know, what is the, then the, the risk profile?
(44:40):
So, these models are text only, by the way.
They are explicitly designed to beused within ag agentic workflows.
So instruction following is a big focus.
Web search, python code execution, that sort of thing.
they've got the ability to providefull chain of thought obviously.
So this is an interesting thing, right?
'cause open ai kind of proprietary models don'tallow you to see the, the full chain thought.
They give you an edited version but not the full thing.
(45:02):
So this is one context where you actuallycan see that because it is open source
architecture wise, we know it's an MOE model.
Obviously 'cause we had the, the model itself they sayit's built upon the GPT two and GPT-3 architecture.
Unclear if that means it is notbuilt upon the GPT-4 architecture.
(45:22):
Interesting, right?
That was not included in that list.
So, so, you know, maybe a, a slight hint that G PT fouris a different beast and we're not actually getting.
A sense of necessarily the architecturalelements that, that make GPT-4 tick in
this release, but still quite interesting.
Two different versions.
I think as you mentioned, a 120 billionparameter and a 20 billion parameter version.
(45:43):
these are MOE models, so they're experts, of course,128 experts in the 120 billion parameter version,
36 experts in the 20 billion parameter version.
And in both cases, you're using the top four.
So in any given token, any given inference run, only fourout of those 36 or 128 ex experts are actually activated.
(46:04):
And anyway, they, they go through a couple of differentsort of decisions that they made on the algorithmic side.
They use banded window attention in alternate layers.
So what this basically means is.
It's kind of like a narrowed context windowwhere each token only gets to attend to
like the say, 128 closest tokens to it.
(46:25):
And this happens every other layer, sothey alternate it with dense attention.
That gives you kind of a, a wider apertureto make sure that there is global information
flowing through the pipes as well.
But this kind of helps you balance just like the computeheaviness of having a full attention mechanism with,
with the need to have that global information processing.
So, group query attention.
(46:45):
So nothing surprising there.
I think you can check out God, the deep seek.
No, well, they did a variant of it anyway.
You can check out our previous podcast on GQA.
And we don't know that much aboutthe training data, by the way.
We train the model they say on a textonly dataset with trillions of tokens.
Okay.
Like, if there is a phrase that could give you lessinformation about the quantity of data used here,
(47:09):
of course it's in the trillions of tokens, right?
That's just the cost of doing business when you'remaking frontier models or anything like this.
The, the question is how many trillions,where specifically did it come from?
They say, well, what the focus onstem coding and general knowledge.
So kinda interesting that we don'tknow the specific number of tokens.
Not sure why, if that's particularly sensitive.
Last piece I'll just mention is on the compute side.
(47:31):
So, they did this with 2.1 billion H 100 hours, and so.
By my back of envelope math depending on thethe precision that they used for training.
And if you assume like 50% utilization, something likethat, you're looking at around 10 to the 25 flops.
So, it's, you know, not a frontier, well, not atrue frontier amount of flops, but it's a really,
(47:58):
like, that's a, that's a highly scaled trainingrun about an order of magnitude shy of what you
see with like, like a grok four, for example.
Yeah, to give you a sense.
So pretty cool.
They, they use deliberative alignment.
Again, you can check out our podcast on deliberativealignment when we first covered it, this is like opening
eyes in a sense, their answer to constitutional ai.
And they do say these models are a lotmore vulnerable to prompt injection attacks
(48:21):
than opening eyes proprietary models.
It's actually pretty wild.
Prompt injection hijacking for these models islike 22% tax success rate for the 120 billion
parameter version, whereas 0 4 mini is like 8%.
So, it's worth flagging that, that thatis a vulnerability with this model.
Just a couple more notes to add to that.
(48:42):
So, on the MOE front the 128 billion parametermodel has a total of 128 total experts.
That means that although having many, many like sixtimes the number of weight compared to 20 billion.
It only has 5.1 billion active parameters per token.
(49:04):
A 20 billion parameter model hasjust a fourth of a total expert.
So it has 3.6 billion active parameters.
What that means is these models are runable on asingle GPU if you have the most expensive GPU around,
if you have an H 100 due to those experts, and dueto doing quantization, that's pretty aggressive.
(49:28):
You can use this on arguably kind of consumerhardware if you're a consumer with a lot of money.
So, yeah, on the whole, a pretty useful model.
I think there is a question mark as to whetherthis will be your model of choice at this point,
or if you would go with Qmi K two or Deep Seeqfor, you know, your fine tuning or, or model usage.
(49:53):
Needs to me still unclear how those options stack up.
Probably on the model size range where this is at, whichis only like medium scale size, right, like Cmic K two,
we saw over a trillion parameters of many experts, Ithink something like 32 billion active founders at a time.
(50:14):
So, as you might expect, thesemodels are pretty good on benchmarks.
They are, you know, capable models, butnot kind of the cutting edge of what you
can get with open source models right now.
Yeah, I, I think the way to think of this is, or one way tothink of it, is it's like an option for western companies
(50:34):
that don't wanna be running Chinese open source modelsso that they have something that's not made in China.
That is, it's a pretty big deal if you're in, inopen source, especially for age agentic models.
'cause you don't know what behaviors havebeen trained in, especially now that, that
CCP is paying really close attention to this.
But yeah.
One other thing too, by the way.
Sorry, actually two quick things.
(50:57):
One of which is they do say that they have explicitlynot put optimization pressure on the chain of
thought for either of these open source models.
So there was a, a open letter or open paper, you canthink of it as that came out a couple weeks ago where
all the big luminaries in the space were saying, guys,can we please, please, please, please not train the chain
(51:18):
of thought to look nice and pretty because we wanna knowif our model is scheming or thinking of evil things.
And if we just sort of like try to safety wash it and, andmake the ugly stuff go away, the model will just learn to
think those thoughts anyway, but just conceal them from us.
And so this is opening eyes saying,Hey, we're gonna walk the walk on this.
And explicitly flagging this paper.
(51:38):
It's an important bit of signalingthey're doing to the ecosystem.
We are not going to put optimization pressure on thechain of thought to make it look pretty for that reason.
Last comment here is on quantization, youmentioned that quantization was a big part of it.
Mixed, floating 0.4 format iswhat they are using for, for this.
That's what's allowed them to compress the modeldown to the point where the smaller one fits on
(51:59):
one GPU, 4.25 bits per parameter is the number.
Basically this means four bits per parameter, andthen there's like one, you have like a block of
parameters that have a, a one scaling factor thatyou use to get the rough order of magnitude, right?
And then, and then your four bits do the do the rest.
But yeah, so essentially.
impressive model that fits right into the ecosystem.
(52:22):
They are open sourcing also a framework called theharmony tokenize that I do expect we'll be seeing more of.
When you make an agentic model, you need newkinds of sort of model self-reference abilities
to refer to different kinds of workflow.
Instead of just like the model andthe user talking to each other.
There's like times when the model needs to think,there's times when the model needs to use certain tools
(52:44):
and so they have a tokenize that explicitly accountsfor that, and they are open sourcing that as well.
So anyway, this is a whole bunch of shit that is droppedat the same time in this paper and could go on for hours.
And certainly opening eye.
Would
Yeah, whole bunch of details.
Last thing to mention as you said, trainfrom a get go to be capable of reasoning.
(53:05):
You can specify reasoning low,medium, high in the prompt, and it's.
guess explicitly optimized to be ableto adjust those reasoning amounts.
Also trained to support tool use out ofthe box and, and it comes with several
tools like web search and python execution.
So yeah, all around you can doa lot of this open source model
(53:30):
onto a slightly more outwear open source release.
We've got Falcon H one, a family of hybrid aheadlanguage models, redefining efficiency and performance.
So this is a hybrid architecture in the sense of combiningtransformer based attention with stake space models
(53:52):
with which we've covered a decent amount over years.
I think definitely there was the timewhen state space models were of a rage.
Quick recap.
State space models are.
A recurrent alternative to transformers.
So you do a sort of loop and can theoretically keepgoing for as many inputs as you want, as opposed to
(54:13):
transformers, which get all the input all at once andkind of get a big chunk of it, and that's all they can do.
You can't sort of keep feeding it data precursor.
So it's been known for some time that it seemsto be the case that going all in on the current
state space just doesn't work as well as goingon a hybrid architecture like you do here.
(54:38):
So this family of models there's a whole bunchof variants with 0.5 billion, 1.5 billion, 3
billion, 7,000,000,030 4 billion parameters.
giant, giant papers, 70 pages whichwe are gonna be able to go through.
Bottom line with 34 billion parameter model is, decent,they don't seem to be competing too much on performance.
(55:04):
If you compare it to QUENT 3 32 B Lama 3.370 B Lama four Scout, has some decent eval
results, but generally not state of art.
So,
yeah, which is what we see, right?
With a lot of these kinda like mambastates-based model type things, right?
They, they make a good proof of concept alongsome axes, and then we just haven't seen them
(55:29):
become like a go-to production grade model.
What is interesting here is the scale.
So it is cool that we're seeing mamba getused at the 30 billion plus parameter level.
That that's cool.
I agree.
It seems so far more on the, I don't callit academic side, but it's it's interesting
that they're pushing this direction.
If Mamba ends up being useful, then they'll havea, a good lead and this is what they have to do.
(55:53):
Right?
So this is you know, the, the Falcon Seriescomes out of TII, so the technology Innovation
Institute in the UAE, and they're playing catch up.
So they, they need a strategy thatallows them to kind of leapfrog.
So I would imagine that's a big part ofthe reason why they're investing in mamba.
It's like kind of the, a good way to learn boththe traditional transformer engineering stuff that
(56:14):
you need to just kind of try to catch up, I guess.
And also to place a bet on maybe we leapfrog becausewe understand how to scale mamba better than other,
or state space models better than other other labs.
That's my rough guess There.
Yeah.
And as with some of our open source releases, a verydetailed report I guess 53 pages, not counting the
(56:35):
appendix going into various details, including quite abit of detail on the training data and some conclusions.
So, you do get faster inference.
They say up to eight times fasterinference in long context scenarios.
And you're able to get better performancewhile using less training data.
(56:56):
So, it's, it's more on the efficiencypiece as per the title of the report.
It also seems to be the case that youget more of a gain at lower scales.
So the 1.5 deep models is competitivewith seven to 10 billion parameter models.
Seems to be, and that's one of the big questions,is if you try to scale up the state space, kind of
(57:22):
hybrid option to the scale of what philanthropicand JGPT are based on, is that gonna be better?
Well, this is not clearly indicating eitherway, but is it showing that, you know,
you get an easy advantage, so to speak?
Onto the next story.
We've got meta clip.
(57:43):
A worldwide scaling recipe.
So this is about contrastive language, image,pre-training clip, which is a classic so to speak.
I think going back to 2022 you are able to inputsome text and some image and see kind of how
(58:06):
they compare more or less the similarity match.
And this is both a model and a bit of a miser scraper.
So perva title, a worldwide scaling recipe,what that's talking about is can you
train on all the languages all at once?
What they say is, if you try to train at a smallerscale, you actually get this curse of MultiLing quality.
(58:33):
So you actually do worse if you trainon both English and non-English.
Data as you scale, you are so to speak, breakingthe curse so you're able to do better by
utilizing all, you know, worldwide data, right?
All the languages without having to translate to English.
(58:55):
And, and that's what they call this meta clip to recipe.
Yeah, I, I am pretty surprised this paper makes no referenceto the idea of like positive transfer and negative transfer.
' it reads to me as like they're trying to coin a phrasereally hard, this idea of the curse of mult ality.
But we already have a term for what this is.
(59:17):
This is just positive transfer, right?
So famously when you, like at small scale traina model on say, three different tasks you add
another task and what you'll find is the model'sperformance on the first three tasks will drop.
And the reason is the model's kinda like overloaded.
It's just like this.
Now it's got another thing tohandle and it's like, ah, crap.
(59:37):
But if you do this at really, really high scales,what you find and with many different modalities,
what you find eventually is that the model actually.
When you give it a new task, itsperformance on the others will go up.
And the reason for that is that it's ableto take the lessons that it's learned from
that new task and apply them to the others.
Sort of like in the same waythat I guess if you let me think.
Well, like, I don't dunno.
Martial arts or something, right?
(59:58):
Let's say like you did Muay Thaiand you did Greco-Roman wrestling.
You're probably gonna be better at picking up Brazilianjiujitsu just because you know how your body moves.
know, you, you, you may have picked up a couple thingsabout controlling someone's upper body from from
Muay Hai, like doing clinches and then for wrestling,like, you know, kind of keeping them pinned, whatever.
That's kind of the idea of what's happening here.
You're just seeing it play out in languages,like what is speaking another language, but just
(01:00:21):
another task that you have to learn to perform.
That's just what this is.
I hate to make it sound reductive, but sometimeslike, think it kind of muddies the water more
because if you actually didn't call this.
Something other than that.
it would invite you more to look at comparisons withlike, how does this work with non-language new tasks?
(01:00:41):
And that's actually almost like a, it's not a missingplot here, but it would be really interesting to see at
least a little discussion of how this contrast, becausespeaking different languages is more similar in task
space than, you know, going from classifying images tocontrolling a robotic arm, but they're on a spectrum.
And it would be really interesting to kind of see thatexplored a bit still a really cool paper and, awesome that,
(01:01:02):
this was done more validation for the idea that scale justautomatically gives you a positive transfer over time.
but yeah, there's just sort oflike a, a little nitpick at my end.
One last open source story, BFL and crea release Flux0.1, CREA and open image model designed for realism.
(01:01:25):
So, we've talked about flux before.
This is basically the best text imagemodel outwear both open sourced and not.
And the big deal here is this is building upon the existingopen source flux variant with a focus on making it so the
AI generated images don't look like AI generated images.
(01:01:47):
It remains to be the case that often you're able to spotAI images just based on them having a certain aesthetic,
which is kind of hard to pin down, but it's something todo with a softness blurriness a smoothness of the image.
It is like surprisingly global.
(01:02:08):
So Korea has a quite long blog post discussing thetopic, talking about how they train it for the images,
looking like real photographs and not clearly AI outputs.
And I think worth noting because yeah, youmight assume you can spot an AI journey.
The image still just based on these aesthetictrends don't assume that to be the case for too long
(01:02:33):
Onto research and advancements.
First story.
Google's newest AI model acts likea satellite to track climate change.
We've got another alpha model from Google.
This one is Alpha Earth Foundations.
It's is designed to track analyze changeson earth using yeah satellite data.
So, a bunch of cameras floating upin space, looking down at earth.
(01:02:58):
And now with this model, users can accessdecal information about any location on Earth.
So you can use this to get at various things.
You can get color coded, map that highlightmaterial properties, vegetation, groundwater
(01:03:19):
human constructions, that less you understandecosystem dynamics, air quality, sunlight.
Et cetera.
And overall this model is intended to assist governments andcorporations in deciding what to do based on understanding
of geography and, and climate and things like that.
It seems like a big part of it is just the compressionthat they apply to the like, terabytes of data
(01:03:46):
that exist about Earth's surface, and just likegetting those wrangled down to a manageable scale.
So it seems like they're overlaying a lot of different,essentially different images of the earth, about
vegetation, about mineral content, about all the things.
I'm not a geologist, but like, you know, all thethings that might be relevant to geography and you
know, research on, on climate and things like that.
(01:04:08):
So, yeah, I mean, there's like limited informationabout how this is done in the article itself.
More that the artifact itself thatyou get it does seem pretty cool.
in the flurry of things this week, haven'thad the chance to look at the actual paper.
So this is maybe one to to bookmark.
We, we probably don't have the capacity to go intofull detail, but as you said, the gist of it is
(01:04:31):
they're compressing a whole bunch of data and,and the challenge where right, is you get a ton
of observations over time from different sensors.
You get noise, you clouds moving in and whatever.
So the model just takes all of thatand makes it usable more easily.
The next one, also a story from Google.
(01:04:54):
They have released Genie Free, an AI modelcapable of generating 3D environments in
real time for user and AI agent interaction.
We've discussed and covered the precursorsto this through two and other ones.
This is another instance of a video generationmodel with realtime responsiveness to input.
(01:05:19):
So essentially you can be an activeagent within an AI generated world.
Some examples of this could be, you know, being a humancharacter in a city environment like you would in, in GTA.
But this is entirely spit out by a neural network.
There's no sort of like, game code or anything like that.
(01:05:41):
No 3D models.
It's all being rendered and.
This one is kind of a big deal.
If you look at example videos,
It's crazy.
extent to which they're able to be consistent over time.
The extent to which like the world doesn'tdistort as you move around or look around,
but instead stage consistent is unprecedented.
(01:06:04):
it's, it's actually quite impressive the, jumpfrom previous models of the store to Genie Free.
We don't know much about it.
They didn't release any sort of research.
It's really just a showing off kind of release.
It's also not usable via demo or anything like that.
But yeah, surprising to me the extentto which this real time attractive model
(01:06:26):
is a leap beyond what we've seen before.
Yeah, so we covered genie.
So this since regenerative interactive environments backin, I, I just went back to look it up before the podcast.
I think it was in February of 2024.
So I recommend checking that out for a breakdownof the architecture or the likely sort of,
at least the, the core of this architecture.
(01:06:48):
It's sort of interesting how they bake in this like, latentaction model that, allows them to train in basically
control you know, like keyboard controls basically.
So you can, you know, move forward, moveback, move to the side, accelerate whatever.
There's like a limited number.
Of actions that you can take in these videos to sort oflike, yeah, generate new frames based on your actions.
(01:07:10):
It's, it's pretty nuts.
And some of the examples too, like you'll see, dependingon how you prompt it, like there was one of like someone
riding a bicycle in a mountain in India and you could sortof see how if you moved your head around you would, you
would like see the person's arms or your arms, I guess.
it sort of extrapolated the body and themechanics of what's happening around.
(01:07:30):
Persistence is a really big, big deal here, right?
So when Jeanie first launched they had coherence timeswhere, as you can imagine, like one example they show is
they, they paint like a blue line on a wall, and then theyturn away from the wall for a couple seconds, and then
they turn back and the blue line is still there, right?
So that persistence, that coherence from frame to frameover long horizons now is on the order of minutes.
(01:07:55):
So the model will remember that that happened by contrast.
the old case, it was, you know, seconds, right?
So you quickly imagine how that makesreality sort of disintegrate fairly quickly.
In this case, you actually have theselong coherence time interactions.
This is really important because it sets up a reallyeffective environments for training agents, right?
(01:08:16):
The big issue with training agents is thatyou need, at least in the real world, is
like the real world takes a lot of time.
It's slow, everything is robotic.
If you move into more like digital settings thanhaving a procedural generation of new environments is
really tricky, especially with realistic environments.
This gives you a way to do that, not forfree, 'cause it costs compute but it gives
(01:08:36):
you a way to generate sort of artificially anddigitally these environments really quickly.
So you can imagine this being useful fortraining a lot of training, agents, quickly.
The other dimension, although youdo have a limited action space.
So you can, you know, maybe have, I guess in the oldversion it was eight different actions that you could
take that they were training the model to, to infer.
(01:08:56):
Here you still have a limited action space, but you alsocan prompt the model in real time to change the environment.
They call this profitable world events.
So you could like prompt in, you know, make analien spacecraft land in front of me, right?
And like now you interact with it.
So it gives you another way to interact beyondthe, the kind of activation space strategy.
(01:09:19):
But they're not yet simulatingother agents in the environment.
they're struggling with creating clear and legible text.
In context other than in, in the text box that you feed in.
So unless you explicitly write the text in yourprompt, the text that you see in the video that's
generated will tend to be kind of garbled and weird.
Anyway, so that's it.
It's a, it's a wild step.
(01:09:40):
I'm visually absolutely stunning.
and this is certainly something thatGoogle has been working on for a long time.
Big advantage to them is that they own YouTube, right?
So they're actually able to train onthat with no issues, other labs can't.
And if it turns out that this kind of environmentgeneration is critical for training agents,
that is a really interesting structuraladvantage for Google and the kind of a GI race.
(01:10:02):
And another advantage is, as you said, DeepMind hasbeen working on this for a long time, setting up like
trainable environments with actual little video games,published research on this, wouldn't be surprised if
that combined with YouTube is what made this possible.
And yet, on the note of realism they're doing seven 20p, so like 70 hd quite sharp at 24 frames per second.
(01:10:27):
So compared to Genie two.
That was, you know, pretty blurry,pretty obviously, ai, looks good.
Like the videos look good.
So very impressive.
Still limited in the sense of how much physics we see.
So, if you try to see like object collisions,there's not a ton of that, but it's getting to
(01:10:50):
a point where you could definitely see yourselfhaving fun playing with these kinds of things.
And next up we've got a paper AlphaGomoment for model architecture discovery.
So the AlphaGo moment here, they makea bit of a major claim in the scraper.
(01:11:11):
So model architecture discovery is apretty well studied topic, basically.
Usually we have just manually.
Chosen how to build new networks, youknow, how many layers to use, what kind of
functions to build with, et cetera, et cetera.
And instead you might consider just directly optimizingthe architecture, doing a search over the space of possible
(01:11:34):
neur network architectures to discover what is ideal.
And there's been, you know, quite a bit of work inthe field, but so far at least it seems that there's
not been a, any sort of revolution as a result.
This paper claims to introduce a system of that leadsto a real kind of breakthrough in model architecture.
(01:11:57):
So if they turn this, their move 37 in modeldesign in the sense that you get a novel
discovery, that is a real game changer.
they get this.
Cream aware router that introduces thequery and summary router that leads to
(01:12:17):
reducing compute while preserving some stuff.
I, I won't get into too much of the details of the discoverybut this came about because of this a SI arch framework
where they have this essentially automated researcher.
it's similar in some ways to what we'veseen before with AI research systems.
(01:12:41):
Like ANA type stuff.
Yeah.
Yeah, exactly.
There's like a whole bunch of stuff going on.
There's a component here of extracting stuff from science.
For papers, there's a researcher agent, an engineeragent, an analyst, and then they take this to explore.
A big tree of architecture is a thousand.
(01:13:04):
773 in a sort of revo evolutionary process.
And yeah, I've seen some disagreement asto how big of a deal this discovery is.
At least some people think that it's a bit of anoverstatement to make this seem like a big deal.
But regardless would imagine your take is partof a thing that people are excited about with
(01:13:28):
regards to AI advancements is self advancement.
When are we gonna get to a point where AIcan do research on its own to get better?
This is at least an attempt atshowing that AI can do that Now.
Yeah, I mean, you're absolutely right.
Like the, this is why there's so much attentionon automated AI r and d in this sort of me sense.
(01:13:50):
This paper?
Yeah.
You know, we're highlighting it because it did the rounds.
It was a viral paper, one of the viral papers of,of last week when there wasn't much to talk about.
I mean, honestly, looking at it like, there wasone bit that caught my eyes, like, I'm not sure
that this is like, as impressive as it seems.
I have, you know, short personaltimelines on when a GI might be hit.
(01:14:12):
But this does not scratch the itch for me personally.
I, so this is a genetic algorithm type of philosophy, right?
That they're applying.
So they take a bunch of innovations that they're researcherand their engineer and all of that, that stuff put together.
And then they'll like they, they'll take elementsof them and kind of mix and match and then
see what the fitness score is based on that.
(01:14:33):
They'll decide which to breed in future generations.
So if you're familiar with genetic algorithms, that's.
That's it.
If you're not familiar just know that like thefitness function is sort of like the core of this.
It, it's sort of the thing you'reoptimizing for in some sense.
Here's what they say.
Architectures with losses greater than 10% belowbaseline are considered to have information
(01:14:56):
leakage and are immediately discarded.
Okay?
So in other words, if you come up with a new architectureand oh shit, it does like suspiciously better than baseline.
In this case they're saying 10%.
Then we go, okay, there must have been someinformation leakage and so let's get rid of it.
This is just buried somewhere in the paper.
Whoa, whoa, whoa.
If we're just like discarding architectures with lossesthat are 10% below baseline because you think they're due
(01:15:22):
to leakage, how do you know there isn't a loss that's likeat 8% or 9% below baseline that's also due to leakage.
And since their big results are reached by iterativelyfollowing the loss through many rounds with each
round potentially being subject to leakage, doesn'tthe whole thing potentially just become a giant
(01:15:43):
leakage stack naively, that's what it seems to me.
Their own assessment suggests.
Like they're obviously not confident in how they'repreventing leakage from happening, which is really hard.
An experiment like this.
I'm not saying that it's 'cause they'reincompetent, it's just that maybe they
bit off more than they could chew, like.
It's just really hard to do, and you probablyought to guess that this is due to like kind of
(01:16:05):
measurement artifacts rather than something, somethingwe'll see may, maybe I'll look like a schmuck.
God knows it wouldn't be the first time.
But that, that kind of was the thing that stuckout to me is like, well, wait a minute, why?
Like why 10%?
How do you really know?
Yeah, exactly.
This was, you're not the only one to have pointedout this little brief detail information that
could presumably meaning, you know, some publishedresearch got through, and that's just what
(01:16:31):
the model did instead of actually being novel.
But yeah, compared to other evolutionary type techniques,I guess we key here is using LLMs to come up with the
code and come up with the novel, directions to pursue.
Yeah, no sort of huge breakthrough in terms of, butpotentially this is another indication of the sort of
(01:16:56):
direction people are taking these kinds of AI research,automated agents where you are trying to combine a
whole system and potentially are gonna be able to kindacreate a research team, so to speak, to make progress.
Nothing huge so far, but you know, GB five is here.
(01:17:17):
Maybe once we get cloud five it'll just work.
Next we have meta evaluates ROC four, another storythat we are kind of catching up on from a week ago.
So, this is on that time horizon evaluation,showing for any given model how long of
(01:17:38):
a task and it reliably, successfully.
And they evaluated graph four, addedit to their kind of trajectory line.
It's able to do slightly more than oh three.
So the 50% time horizon the time market, which Rockportis capable of doing the thing that takes that long
(01:18:01):
50% of the time is now about one hour and 50 minuteswhich is up from all three at about an hour and a half.
So, this is you know, yeah, another kindof leap aside advancements, at least at to
50% time horizon for the 80% time horizon.
(01:18:24):
It actually doesn't make advancementsbeyond O three and Opus Sport.
Oh, I, I think that's fascinating, right?
I mean, so I was thoroughly confused about this, and this isone of the reasons that I was so excited about the GPT five
launch is like, we'll get another point on that meter graph.
we'll be able to see, does this one look as good on the 80%?
So that one does.
(01:18:44):
So g PT five continues the trend through 80%.
It's consistent with the sort of acceleratedtimeline with four month doubling time
and all that jazz that we talked about.
But interestingly, grok board doesn't, and th that I thinkis worth, in a weird way, it's a shame that we didn't pause
longer on Grok four be before getting GPT five, because Ithink it would've caused a really interesting conversation
(01:19:05):
where people would go, so which one matters more?
Do we care more about the 50% orthe 80% replication probability?
And I'm not sure what the right answer is there.
But yeah, sorry.
There's like a whole podcast we could do about there the distinction between those in White matters.
Bottom line is we don't have to do that now'cause GVD five gave us our, our extra points.
(01:19:25):
So there you go.
Yeah, we we're just continuing to update this chart.
I dunno what you can probably searchup when we initially discussed this.
It's an interesting effort at trying to quantify basicallythe real world impact or potential usage of these models.
GB five, they say is able to do two hoursand 20 minutes, something like that.
(01:19:50):
At 50% success rates for 80%.
It's a bit higher, 26 minutes.
So as you said now leading the pack quite a bithigher than GR four or O three a cloud four sonnet.
moving on to policy and safety.
First, we have a paper from OpenAI estimatingworst case, frontier Risks of Open Weight LLMs.
(01:20:15):
So they study the worst case risk of releasing G-P-T-O-S-S,and they talk about particularly malicious fine tuning,
where they attempt to elicit maximum capabilities byfine tuning G-P-T-O-S-S to be as capable as possible
in, bad biology stuff and bad cybersecurity stuff.
(01:20:40):
So, bio risk where they are trainingin an RL environment with web browsing.
And then for cybersecurity, they're training in anagent coding environment to solve capture of a flag.
The gist is that G-P-D-O-S-S may slightlyincrease capabilities, but doesn't substantially
(01:21:02):
increase the risk factor of lik survey.
The decision to release model is informed by theidea that compared to other open source models
out there, this is not gonna significantly changewhat bad people might be able to do with it.
and then I, I saw some rebuttal as well.
People saying basically like, if, if the theoryof the case is that we're going to allow ourselves
(01:21:25):
to release a model, as long as it only slightlyincreases bad actors' ability to do damage, then this
creates a situation where all the frontier labs areincentivized to incrementally increase the capabilities,
the weaponized capabilities of open source models.
I mean, this is true.
It's, this is a very difficult hair to split.
(01:21:46):
Some might not see it as a hair, but it certainly is.
Like, you know, there is nuance here, part of which islike you have these open source Chinese models right
now that are the only ones on the market as open source.
Increasingly, those are age agentic models,which means people will be deploying them in.
Settings, potentially with dangerous behaviors baked intothem that adversaries want to manifest in certain ways.
(01:22:09):
So anyway that's kind of all partof the debate back and forth there.
Yeah, it, it is by the way, pretty pretty impressiveon these benchmarks, including, in particular,
bio risk tat knowledge and troubleshooting.
It is essentially beats so, beats perplexity deepresearch which is based on R one plus browsing.
pretty close to opening IO threewith anti refusal and, and browsing.
(01:22:31):
And by the way the two things that they tryhere when they do their malicious fine tuning
this MFT thing, they're sort of, coining here.
One is.
You can disable refusals generallyand just generally jailbreak a model.
The second is domain specific capability maximizationor just basically fine tuning the model on the a,
the dangerous capability you want it to develop.
(01:22:52):
And these are two different kinds of finetuning that they have to account for, right?
Get rid of the model's tendency to just declineto help you with bad things, and then separately
make the model better at doing bad things.
So all part of this sort of framework that you see themtrying to sharpen here as they work towards implementing
what is ultimately in their preparedness framework, right?
This is a, a step that they're tying to that.
So there you go.
(01:23:14):
Next one and another safety related piece of research.
This one from Enro covered in some mainstream articles.
The article title is Ros ai vaccine,train it with Evil to Make It Good.
That's with FunTech on the actual researchtitled Persona Vectors Monitoring and
(01:23:36):
Controlling Character Traits in Language Models.
So, so this is actually kind of related to someof the research I think from OpenAI recovered
just recently that looked into this notion ofyeah, basically traits that models have in, in
terms of being what, sarcastic or dishonest.
(01:23:56):
And at the time, what we showed with this previous researchis that if you just increase being sarcastic somehow,
that makes you also misaligned in all sorts of ways.
So this research is looking at monitoring thepersonas that a model has and then mitigating
(01:24:17):
the ability to fine tune personas that are bad.
And yeah, basically this, this talks about thepipeline of data that is being used and would
make it so it's, it's harder to misalign theentire model with just a, a bit of tweaking.
(01:24:37):
I actually again, week of week of insane Stuff blasting.
I did not see this paper and I'm just looking at theBusiness Insider article about the paper, which is
really confusing and use it as very kind of like, it'svery journalist trying to write layman language vibes.
But reading between the lines and, and we'll correctthis if this is wrong next time, my sense is what's
(01:24:59):
happening is that basically anthropic like putsin a prompt and they get, you know, obviously
the activations in the model that, that result.
Then they take up one layer probably in the residual streamand they would inject the evil persona that they have
derived from using their sparse auto encoder strategy,which we talked about before and can talk about later.
(01:25:20):
If people read in that they wanna hear more aboutthis, but basically inject the evil persona, like
add it at that layer but keep the training objectivesuch that like, hey, be, be harmless basically.
So that even with that injected activationthat artificially makes, at least as of
that layer makes the residual stream evil.
The rest of the model learns to compensate for that.
(01:25:43):
so when you then remove it, essentially the model is.
Like it's forced to behave well under, under thecircumstance where you inject evil into its brain.
So it's like if I took a slice of Andre's brain and Iput in like an evil slice of my brain, and, but then
like, I don't know, I, I gave Andre some like reeducationtraining to be nice and then I took out the evil slice
(01:26:05):
of my brain and Andre's like, oh, I'm extra nice now.
And even if somebody put an evil sliceof brain in me, I would still be nice.
That's the vibe.
Again, just my guess based onthis, pretty hard to parse article.
Yeah, to be honest, that was kind of what I was doing.
As per usual, trying to make it sound likeI actually understand diving a bit deeper.
(01:26:26):
The focus of a paper itself is more on monitoring as well as
Oh,
So, they introduce persona vectors.
That's the kind of activation vector thatindicates a model has a certain persona.
And then what that means is you are ableto monitor the model during training for
(01:26:47):
the emergence of this kind of personality.
And then also monitor the fluctuations of a modeladd deployment time, which means that you can
then mitigate that happening by flagging trainingdata preventing it in first place, or intervening
when stuff goes wrong, when you're able to detect.
(01:27:09):
So this, this basically tells you another way todetect misalignment in, I guess a more reliable way.
Well, now I'm curious about the thing that I said andwhether someone's gonna try that, because that sounds
actually kind of interesting and I wonder what would break.
Yeah.
Yeah.
And next up, another result from Monro.
(01:27:30):
And again, we are not as read on this.
This is slightly older.
This is going back two weeks, butI don't believe we've covered it.
So I think it's worth mentioning the research is titledBuilding and Evaluating Alignment Auditing Agents.
The headline is they develop free agents thatautonomously perform alignment auditing tasks.
(01:27:53):
And when these models are tested against intentionallymisaligned models, these auditing agents are
successfully able to uncover that problem, right?
And so.
This is pretty much what it sounds like.
You know, if you worry about misalignment, ifyou worry about models being bad, one way to
(01:28:16):
try and prevent that is to have some other aido a bunch of tests automatically to detect it.
This is Aaro taking a stab at it and showing thatat least to some extent you can make that work.
On the lightning round we have this researchnote, so not a paper, but, and again, even papers
(01:28:37):
aren't papers, so let's just call it a paper.
Optimizing the final output can obfuscate chains of thought.
So.
This is by I think Alex Turner was, was part of this.
He was like one of the original guyswho did research on Yeah, there he is.
Yeah.
Sorry.
Turner Trout is his, his handle, but it'sAlex Turner and a bunch of other, other folks.
He's one of the original researchers on the power seekingresearch that we may have talked about a while ago.
(01:29:01):
it turns out that if you take a model andtrain it to for example, I mean they so
they, they train a model to, what's the word?
Oh yeah.
Okay.
I'll give you a task.
Sorry, I'll start with the task.
That description.
Give a model description of different objects, likea shiny wooden cube or something, and tell the model
(01:29:23):
whether or not that object triggers a detector.
And then you're gonna ta, you're gonna give a bunchof examples of that to the model and essentially
have it become a classifier, have it predict, givena new object, whether the detector will be triggered.
But now what you do is you penalize the modelfor using the word detector in its final output.
(01:29:45):
All you care about is the finaloutput, not the chain of thought.
You penalize the model for using theword detector in the final output.
What you'll find is that the model uses the worddetector less in its chain of thought, which
is interesting because you never use the chainof thought as an input to your reward, right?
So, they're calling this spillover.
(01:30:05):
Basically the thing that you are optimizingfor the property that you're optimizing for in
the part of the output that you're evaluating.
Spills over into the part of theoutput that you are not evaluated.
And they do this in a bunch of differentsettings and not worth going over the details.
so, so super interesting.
One of the implications here is we wantto like, ideally not optimize the chain of
(01:30:31):
thought to look pretty or nice or whatever.
Because that will tend to cause models to obfuscate ifthey're, if they're hatching evil plans, we want them to
write those evil plans in plain English for us all to see.
So one solution people had proposed is, okay, fine, well,let's just only use the actual output as the thing that we
reward and reinforce on and not use the chain of thought.
(01:30:52):
So the chain of thought can, can, you know,be free and, and honest and all that stuff.
This is showing you that that actually isnot sufficient or may not be sufficient.
It's also testing a proposal for analignment strategy that Dan Cota proposed.
think a couple months ago, which is really cool,called Mind plus Face which we should talk about at
some later point because we're running out of time.
(01:31:14):
Good.
I'm running out of time.
Lots to talk about.
So, moving on to another one of the, our favorite topics
compute and China.
Got a couple stories on that.
First up from Epic ai, why China isn'tabout leap ahead of the West on compute.
(01:31:35):
Basically what it sounds like, this is more of ananalysis piece than news that covers the various
factors, the manufacturing side, the software side ofwhy China is not likely to overtake the West in terms
of the frontier of let's say like removing the needfor Nvidia to be on the cutting edge of AI compute.
(01:32:00):
It's, it's fairly detail piece that coversprobably what listeners regular listeners
already know in terms of the factors at.
But yeah, if you're interested in, in getting abunch of details and getting into slightly more nitty
gritty of things like Cuda, this article has that.
It's a shame 'cause this is one of those articles where likethe actual value is in the details, and then the superficial
(01:32:24):
description is gonna be like, ah, you already know this.
The, the one thing I'll say is that likemost epic AI research one of the most
interesting things about it is that they.
Bother to plot and, and analyze the, like, doublingtimes or, or like how, how quickly, like flop per
second or like compute power is growing over time.
how their compute efficiency is growing over time.
(01:32:45):
Comparing it to how compute efficiency hasbeen growing over time in the West, and kind of
like drawing lines and having them intersect.
So if you're, if you're into that kind of stuffas I am, you'll find this really exciting.
You won't find anything qualitatively surprising here.
Photolithography is the big thing that China's missing.
No surprise.
They're making progress on everything.
That's not that, that it'll take a coupleyears before that, that really has an effect.
(01:33:08):
And, you know, probably we shouldn't beexporting Nvidia chips to China as a result.
It's sort of the subtext.
And next story inside the summit whereChina pitched its AI agenda to the world.
So this is covering the release of the Global AIGovernance Action Plan from late July, which happened
(01:33:33):
at the World Artificial Intelligence Conference inShanghai, which I wasn't aware of personally as.
A, as a thing that happened, but apparentlyit's a major event for ai kind of personalities
to gather including major prominent westernfigures like Geoffrey Hinton and Eric Schmidt.
(01:33:57):
And the gist is that China, at least in thestatements and this action plan very much advocated
for cooperation coalition of major AI safetyplayers call it by China Singapore Que and vu.
Yeah, in contrast to the US with the US AI actionplan, kind of, you know, going the opposite way, very
(01:34:25):
much going us as a loan figure in the space, yeah,it, it seems to be, I don't know, it's hard for me
to understand how meaningful this is, but there isa contrast to be seen between the US AI action plan
and this global AI governance action plan Clearly.
So the USAI action plan often is kind of likemischaracterized as not caring about safety, whereas what
(01:34:53):
they do is actually like, they call for investments in.
Basically AI lender research, AI control and monitoringmechanisms to detect if things are going haywire and
contingency plans as well for what to do if they do.
So this is like, it's not agovernment that's going, eh, whatever.
It's a government that you can model as saying, well,like, we're skeptical, we don't know, but we're maintaining
(01:35:16):
intellectual sort of I don't know if intellectual humilityis the right term, but like we're, we're uncertain enough
that we're gonna keep our, finger on the pulse here.
There wasn't, so there was some internationalengagement stuff on the chip side in the action plan.
There wasn't much in the way of like,hey, let's coordinate around safety.
And that's where this sort of sentence fromthis article comes in when they say with the
(01:35:38):
US out of the picture, because there was nosort of senior representation from America.
There.
Quotes, a coalition of major AI safetyplayers co-led by China, Singapore, the uk.
The, and the EU will now drive efforts to constructguardrails around frontier AI model development.
Now, it's not clear that that actually is meaningfulgiven China's involvement and also the fact that
(01:36:01):
the United States, if you ignore their labs kindof is, like the most important player, arguably.
So apparently also, it's not just the fact thatthe US government was not there, there also wasn't
representation from the leading frontier Labsonly XI sent employees to the forum, which I mean.
If I heard that a bunch of like open AI andphilanthropic people were there, I might be
(01:36:25):
slightly worried that they didn't all bring burnerphones and laptops because this is like China.
but you know, it seems like this is this is akind of China sphere sort of validation piece.
Another piece there is this guy Brian say whois one of the, the founders of Concordia ai.
This is like a Beijing based safety like research company.
(01:36:47):
And he has been in this for a long time.
I remember talking to him like years ago, just whenI was trying to understand the lay of the land.
so they're claiming many Western visitors weresurprised to learn how much of the conversation
about AI in China revolves around safety regulations.
Quote, you could literally attend.
AI safety events nonstop in the last seven days.
And that was not the case with someof the other global AI summits.
(01:37:09):
Part of this is also that China has aparticular interest in controlling the
outputs of these models for political reasons.
A kind of zero tolerance for referencesto Chenin Square or the Uyghur situation.
and so that kind of creates more of an imp an impetusfor them to cla down on the narrow behaviors they don't
like while making no mistake, having absolutely noguardrails whatsoever for the CCPs own use in government,
(01:37:36):
in defense, in kind of national security of these models.
And that's one sort of distinction thatthey do weaponize quite effectively.
They'll be like, Hey, we're imposing all these like safetyguardrails on our models for our domestic companies.
There are folks in the West who get excited aboutthis and go, oh, China's serious about safety.
But of course, in China, what mattersis how will the government use it?
(01:37:59):
And you'd better believe there's not gonna beguardrails on government use of the, the technology.
So it's a bit complex.
But you know, people are talking about it.
I don't think that's a bad thing.
Just gotta keep the caveats in mind.
Right.
And, and there's no meaningfulkind of updates in terms of events.
But it does speak, I think, to positioning framing.
(01:38:20):
Lately it's been an interesting development thisyear with China being the source of the leading open
source models now you know, basically in the west.
And this kind of political effort on engaging with otherinstitutions and then collaborating and so on, is another
potential effort to sort of, have a sphere of influencein AI beyond being a singular kind of closed off player.
(01:38:48):
Last story, of course, we've gotta talkabout export bans and stuff like that.
Nvidia, H 20 GPUs are portally caught up in the US CommerceDepartment's worst expert license backlog in 30 years.
Billions of dollars were for GPUs and other stuff arein limbo due to staffing cuts and communication issues.
(01:39:13):
So, that's very long title, prettymuch telling you what's going on.
There is yeah, a real delay in, in difficultyin shipping with chips and particularly not
just to China, but other regions as there's theneed for approval for these expert licenses.
(01:39:36):
Yeah, my personal take on this is, this isactually a, a mistake in the first place for us to
be shipping GPUs to China, especially given stufflike what, what Epic surfaced in their report.
What, what everybody's basicallyflagging is like China actually cannot.
Or has no line of sight on, on catching up to
On video to the west.
Yeah.
(01:39:56):
exactly.
That's it.
And so, the, the frame has kind of been, well, solet's ship China GPUs that are like considerably
worse than the GPUs that we can use here.
But I mean, in my opinion, the frame oughtto be let's ship China GPUs that are just
marginally better than their best domestic GPUs.
(01:40:20):
It should be more of a, what ceilingthan a floor or floor ceiling.
I don't know, something like that.
And that's, that's just not what's happening rightnow with the age 20, which I think is a shame.
The other thing is that like.
China could even fab like effective systems like CloudMatrix and that, but their challenge is doing it at scale.
So they literally can't, even if Nvidia, like H 20GPUs, were a lot shittier than they are, they just are
(01:40:42):
benefiting from the excess capacity and having thesethings fab, for example, like TSMC and, and not SMIC.
So I think, I think those are like real issues.
I think this is a, it's a real shame, but I understand why.
So the theory here is something like, we don'twant Huawei and SMIC to then have dominant market
(01:41:02):
share in China and then eventually ramp up capacityto be able to compete within Nvidia elsewhere.
The thing is that's, I don't thinkat risk of happening anytime soon.
And historically, the way the Chinese supply chain hasworked is we've done things like block off narrow parts
of their supply chain but not others, which creates liketons of economic pressure on both sides of the supply
(01:41:23):
chain to support the closing of that gap when there's justone part of your supply chain that you're missing, you
have buyers and sellers on either side who are willingto like, help co-invest to build up that ecosystem.
Whereas if you just nu the whole supply chainand like carpet bond it which arguably is what we
should have been doing like starting five years agothen it just creates a much harder circumstance.
(01:41:45):
I don't think that there's a world where Chinacould possibly be putting the pedal to the
metal harder on domesticating their AI supply chain anyway.
So it's not like us selling them Nvidia chips in anyway, in any way, reduces the amount of dollars or effort
that is going into the SMIC Huawei kind of complex.
So I don't see a policy win frankly, in this.
(01:42:08):
Again, I understand the reasoning behind it.
I just don't see this materializingin the way that is hoped for.
But maybe I'm wrong and we'll just have to wait and see.
It, it also doesn't seem like it's in anysort of intentional, this intecho thing.
This is kind showcasing the stateof the US bureaucracy more so to me.
(01:42:29):
Yeah.
This particular thing.
Yeah, you're right.
It's just like bureaucratic incompetence, like failing to,
yeah.
I did wanna address one listener comment real quick.
There was a review on Apple Podcast titled ThePolitics of the Internet where it basically, it
begins with, have to disagree with your discussionof internet data causing chatbots to lean liberal.
(01:42:55):
And then it basically goes into variouskind of topics related to what typically
Democrats versus Republicans in the US say.
So I just wanna clarify when we say.
Chatbots, lean liberal.
This is drawing on studies that show that if you askchatbots various questions and see what sort of ethical
(01:43:20):
and moral values they align with, they tend to go moresort of Democrats friendly, even Republican friendly.
This is like well known.
When you say bias, it doesn't meannecessarily that it's bad or wrong.
This is just how these models tend to behave.
They're more in line of Democrats.
(01:43:41):
So this is not to say that they say false swings or, youknow, are, are somehow bad, but they are gonna agree with
Democrats more than they are gonna agree with Republicans.
It, it is something that we have to like face, honestly.
Like there's a lot of studies that, thathave been done, including from sort of
(01:44:02):
organizations that Lean left a little bit more.
There's there was a famous study, this is an organizationthat Le Leans left, but the Center for AI Safety did
a really big one that was pretty decisive in termsof carving up, like, you know, these models very
clearly seem to, and they had really good methodology.
We, we covered the paper on the podcast actually, but abunch of others have come out since, and it's, it's pretty
(01:44:23):
obvious too if you just interact with the chatbots, you'llsort of see like, there are all these examples of, you
know, if you ask a question about Hillary Clinton versusDonald Trump back in the day this is just like, not the labs
trying to do this necessarily, as you said, it's just like.
Just think for a second about thedemographic of who's online, right?
Skews younger skews towards people who live in big cities.
(01:44:44):
Well, statistically that means quiteoverwhelmingly Democrat voters, right?
So, it's, it's just a function of whereyour data comes from and that makes it, by
the way, incredibly hard to come up with.
I mean, we don't know how to make anunbiased reporter as a human being, like
that's not something anyone knows how to do.
So you can imagine how impossibleit is to do with AI systems
that it's just kinda like what
(01:45:06):
think it's a good point, right?
Like if, if you try to make it perfectly,perfectly neutral, then it's just gonna
refuse to say anything on most topics, right?
So, again, bias is usually considered a negative term.
This is more of a descriptive term, you could argue.
This comment also talked about GR a bit, so thisis drawing on the whole conversation about grok.
(01:45:28):
To be clear, I don't know if this was obvious, Iam quite critical of the way that Grok has handled
misinformation or, being maximally truthful.
They've done some things thatare questionable in that regard.
So didn't want to imply that grok isaddressing a problem by kind of correcting it,
(01:45:50):
Yeah, you,
different thing,
you can think of, yeah, all these models arejust applying, or these labs are applying like a
different philosophy in a different lens, right?
Like ultimately we don't know what theone true way to do this is gonna be.
And to some degree, maybe having differentlenses on it is probably a good thing.
And the same way that you don't wantto consume all your media from like.
(01:46:11):
either like Fox News or CNN, youwant to like have a nice balance.
Right.
Sort of similar idea here.
There's, because there is no answer, theonly answer is the meta answer of, have a
balanced media diet or a balanced LLM diet.
I don't know, that's a half bake thought,but like, this is just a really hard problem.
It's Harder than it seems.
Well, that's the episode.
An exciting week and we'll try to, Stick around nextweek as well, not skip any more for the next while.
(01:46:38):
Thank you so much for listening.
As always, we appreciate it.
If you give us feedback, if you share the podcast allof that, more than anything, please do keep tuning in.
Popular Podcasts
New Heights with Jason & Travis Kelce
Football’s funniest family duo — Jason Kelce of the Philadelphia Eagles and Travis Kelce of the Kansas City Chiefs — team up to provide next-level access to life in the league as it unfolds. The two brothers and Super Bowl champions drop weekly insights about the weekly slate of games and share their INSIDE perspectives on trending NFL news and sports headlines. They also endlessly rag on each other as brothers do, chat the latest in pop culture and welcome some very popular and well-known friends to chat with them. Check out new episodes every Wednesday. Follow New Heights on the Wondery App, YouTube or wherever you get your podcasts. You can listen to new episodes early and ad-free, and get exclusive content on Wondery+. Join Wondery+ in the Wondery App, Apple Podcasts or Spotify. And join our new membership for a unique fan experience by going to the New Heights YouTube channel now!
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.