#202 - Qwen-32B, Anthropic's $3.5 billion, LLM Cognitive Behaviors - Last Week in AI

All Episodes

#202 - Qwen-32B, Anthropic's $3.5 billion, LLM Cognitive Behaviors

March 9, 2025 • 79 mins

Our 202nd episode with a summary and discussion of last week's big AI news! Recorded on 03/07/2025

Hosted by Andrey Kurenkov and Jeremie Harris. Feel free to email us your questions and feedback at contact@lastweekinai.com and/or hello@gladstone.ai

Read out our text newsletter and comment on the podcast at https://lastweekin.ai/.

Join our Discord here! https://discord.gg/nTyezGSKwP

In this episode:

Alibaba released Qwen-32B, their latest reasoning model, on par with leading models like DeepMind’s R1.
Anthropic raised $3.5 billion in a funding round, valuing the company at $61.5 billion, solidifying its position as a key competitor to OpenAI.
DeepMind introduced BigBench Extra Hard, a more challenging benchmark to evaluate the reasoning capabilities of large language models.
Reinforcement Learning pioneers Andrew Bartow and Rich Sutton were awarded the prestigious Turing Award for their contributions to the field.

Timestamps + Links:

cle picks:

(00:00:00) Intro / Banter
(00:01:41) Episode Preview
(00:02:50) GPT-4.5 Discussion

(00:33:19) A.I. Start-Up Anthropic Closes Deal That Values It at $61.5 Billion
(00:35:49) Nvidia-Backed CoreWeave Files for IPO, Shows Growing Revenue
(00:38:05) Waymo and Uber's Austin robotaxi expansion begins today
(00:38:54) UK competition watchdog drops Microsoft-OpenAI probe
(00:41:17) Scale AI announces multimillion-dollar defense deal, a major step in U.S. military automation

(00:44:43) DeepSeek Open Source Week: A Complete Summary
(00:45:25) DeepSeek AI Releases DualPipe: A Bidirectional Pipeline Parallelism Algorithm for Computation-Communication Overlap in V3/R1 Training
(00:53:00) Physical Intelligence open-sources Pi0 robotics foundation model
(00:54:23) BIG-Bench Extra Hard

(00:56:10)
As usual.
In this episode B, you'll be summarizing and discussingsome of last week's most interesting AI news.
You can go to episode description or to last week in ai.com toget the links to all those new stories and the timestamps as well.

(00:32):
I'm one of your hosts, Andre Kko.
I studied AI in grad school and now work at the generativeAI startup startup aade, which someone on Discord mentioned.
I never actually saved an ai.
Oh yeah, it's true.
Yeah.
Okay.
We don't have much much publicly out there, which is why I don't say a name.
But it's coming so soon.
We will have a bit of news of our own to share, which will be exciting.

(00:55):
Very cool.
Okay.
Well good to know.
And, and yeah, I'm Jeremy Harris, the other co-host,Co-founder of Gladstone ai, AI national Security company.
And I've been away for, is it three weeks?
It's gonna be a couple weeks.
A few weeks, yeah, three weeks.
it's been a wild time.
We're, we're gonna be launching something in a, mm, actually it'll be like April 20th now.

(01:15):
We just had it pushed back for reasons.
you'll, you'll hopefully learn more about why I've been out of pocket this whole time.
And hopefully it'll be worth it.
We'll see.
And anyway, yeah.
Really excited to be back in the saddle.
It's been a, an interesting week, right?
I mean, there's like, not the huge, I mean, the last weekwas the big, big week for new model releases this week.

(01:35):
I think some, some specific really interesting and impactful things, but not a huge number of them.
Yeah.
To give you a quick preview of what we'll have in this episode, we'll have onenew exciting model from Alibaba this time that pretty much caps off a trend,
I would say, of new reasoning models, new big models, new cutting edge models.

(01:56):
And then aside from that, we'll be going to kind of going back to a mix of usual storiesof some new consumer facing things, Alexa, plus a few kind of business deals going on.
Quite a few open source releases.
We were focused on benchmarks and also some very cool unusual kind of software from deep seek.

(02:18):
Then in coding event in research will be once again talking about reasoning, whichhas been kind of a main, major focus for research for a while, but also about
honesty and accuracy, which will be interesting, and policy and safety will be mainlygoing into some of the details of the new models, and of course a little bit about.

(02:39):
Export controls as well.
But before we get there, we were thinking since Jeremy, you could notco-host the last week where we were discussing GT 4.5 as well as cloud free.
And then also, if we could do a quick section to chat a little bit more aboutthose releases and, and probably the more interesting one is GP 4.5 with kind of

(03:03):
the narrative going on that it seems kind of underwhelming and it seems like maybeit's showing the limits of pure unsupervised scaling, where, you know, reasoning
is now the thing that's getting us really, really big leaps in performance.
Unclear if unsupervised reasoning by itself is, is really worth for investment at this point.

(03:26):
Yeah.
Th this is really interesting 'cause this comes up every, with everybeat and I think there, there's a little bit of nuance that's missing.
So first of all, we gotta talk about this idea of code smell, because, sorry, not code smell.
Sorry, of of model smell.
Yeah.
It used to be code smell back in the day.
I'm old enough to remember 20 minutes ago when the kind of smell you cared about was code smell.

(03:47):
But yeah, so, so essentially we live in a world where with every beat of pre-trainingcompute, so with every new 10 x like order of magnitude increase, and the number of, of
flops that are thrown at a system during pre-training You are getting a better model.
It used to be that you would get a very noticeably better model, right?
So GPT two, would barely be able to string together three coherent sentences.

(04:10):
Then G PT three I think that was more like a hundred x the compute, but stillget the gist all of a sudden is like, you know, full paragraphs in in essays.
And then G PT four gets you to kinda like page length content and beyond.
You can kind of clearly tell what's G PT two, what's GP three?
What's g PT four?
The problem is, we've talked about this in the context of image generation systems, right?

(04:32):
There comes a point where it's just really hard to tell what is better, right?
Like show me the absolute latest from Mid Journey, show me theabsolute latest from really like any image generation system.
They, they're all.
Basically photorealistic.
You have to be really, really specialized to be able tokind of catch the smell, if you will, of these models.

(04:52):
Hence this idea of smell, right?
This very kind of esoteric abstract notion that I can't put my fingeron it quite, but if I play with the model enough, I can sort of tell
that there's more intelligence or capability under the hood, right?
And so, yeah, exactly.
And we, we were talking, there's smell, there's also kind of the vibes, and that'soften when you model release you know, you look at the benchmarks usually, you know,

(05:15):
you get some numbers, but it's really hard to tell what those benchmarks even mean.
So often you want to just go see what the vibes are from trying it from, for instance,like Andre Kafi, posted evaluations of gr free of GP 4.5, and, and gave a stake on it.
So that's a similar aspect where you need a very, very specialized benchmark Yeah.

(05:39):
To showcase the difference.
And you need just firsthand experience in reporting on trying side by sidethe difference between models, which, which also makes a ton of sense, right?
Because fundamentally, pre-training is about what, it's aboutbeing able to predict the next token with really high accuracy.
Right?
And what that means is once you get really, really, reallygood at that, you, you must have learned a lot about the world.

(06:02):
Like being good at fill in the blanks means having a good world model ultimately.
At least that's the thesis.
And so the, the challenge is like.
Where does your, where does your lift come from?
Where does your performance improvement come from on that metric?
Once you get to GT four?
Once you get to GT four?
Oh, well, it's gotta come from getting more and more niche things, right?
It's gotta come from like getting better and better at, I don't know, like biosomething or like kind of niche fields in history or, or shit like that, right?

(06:30):
So this is one dimension of it.
Another is maybe you get a little bit better at logic and induction and, and all that stuff.
And so the interesting thing is that.
We're, we're getting to the point where it's just hard to, to almostqualitatively gauge just by reading the outputs of these models.
I will say quantitatively, when you look at Chatbot Arena, you know,there it is, GPT-4 0.5, right at the top, along with Glock, Glock three.

(06:53):
So when people are actually kind of at scale asked to assess the performance,these systems in many contexts, they can tell the difference there.
And, and you see readouts from Carpathia and others who kind of allude to similar things.
But I think the main value in pre-training now is not just the model that you get.
And, and this is something that's shifted.
This was the reason by the way, that on the podcast we so easilymade the call when we were talking about deep Seek V three, we said.

(07:18):
Big reasoning breakthrough is coming from deep seek.
That was trivial.
That was the easiest call in the world.
No one was paying attention to it at the time, but you could tell howgood the reasoner would be based on the strength of the base model.
The base model wasn't that much better, like on a percentage basis relative to other base models.
'cause it gets harder and harder right as you get closerand closer to a hundred percent on these benchmarks.

(07:38):
Yeah, you're like climbing that next percent meanslike kind of understanding a lot more about the world.
And anyway, so that's all kind of part of, part of this.
And it creates the illusion that you have a system that's sort of saturatingand, and not giving you any ROI but then when you apply the right amount
of inference, time compute, suddenly you unlock this, this massive advance.
the last thing I I wanna flag here is really this intuition.

(08:01):
It's a bit of a toy picture and it's not perfect, but if I give you,you know, a hundred hours to spend on a test you have the choice.
You know, do you spend you know, 99 and a half ofthose hours studying and half an hour doing the test?
Or do you spend 80 hours studying and 20 hours doing the test?
The models, the base models like V three, like GPD 4.5, essentially are just spending.

(08:27):
Like, you know, all but one minute all but one second studying for the testand then they're spending one second actually performing in, in a test time.
Right?
That's what a base model really is, and no surprise you saturate that at a certain point.
There's only so much more leverage you can get out out of your pre-training, but then suddenlywhen you open up, when you allow the model to reason a test, like, damn, what a, what a lift.

(08:50):
And one of the big discoveries in the last few months and arguably years.
Has been that really scaling doesn't mean just jacking up your pre-training compute.
It means jacking it up in tandem with your inferencetime, compute budget and moving them up together.
There are curves essentially that you're jumping throughas you do that, and that is what continues ad infinitum.

(09:10):
And so this can lead or, or which vary, well, may I should say that leads a lotof people to, to make the, I think, pretty clearly incorrect assessment that
Oh, the, you know, we're saturating scaling in the general sense when reallywhat's happening is, yeah, if you don't do any inference time compute Yeah.
You're pre-training, you're, you're basically like you've doneas much as you can with, with, you know, studying for the test.

(09:33):
It's now time to spend more time actually doing the test.
So I think that's kind of the, the, the big thing here.
It's not a coincidence that OpenAI was so confident insaying G PT 4.5 will be our last our last base model.
Basically our, our last auto aggressive straight up, you know, text auto complete model.
From now on, it's gonna be reasoning models.
Why?
Because the economics don't support spending more time on just pre-training and not on inference.

(09:57):
The scaling curves are a combination of inference and training time.
So I guess that's my, overall take on this stuff.
I think we'll, we'll continue to see improvements.
I really would caution against betting on the scalingtrend just based on, on, you know, GPT-4 by five.
Well, I think there's a bit of nuance and, and we'll move on, butI'll just add a couple things 'cause this is an interesting topic.

(10:20):
I guess a big question is.
In terms of scaling up model sizes, are we kind of hitting the ceiling?
Right?
And that is not really known.
I would say like GB 4.5 is one indicator of that.
We don't know the model size.
People are guessing.
Maybe it's another 10 x step where it's like 10 trillion weights.

(10:40):
We have no idea.
But based on the cost, based on the speed of it, it seemsto be a much bigger model with many, many more weights.
And doing that resulted in these relatively incremental changes from GP four point four as you said.
Which does kind of make sense because there is kind of a difficulty once you'reintelligent generally in terms of answering, you know, opening, I was highlighting this

(11:05):
like emotional intelligence aspect, which arguably could mean that, you know, it is.
Much better at kind of the nuanced reasoning of how to reply to agiven question in terms of tone, in terms of depth of response detail.
But those are the sorts of things that you can't benchmark.
So that's one thing.
And then the other thing is, as you said, we found that with base models, withjust a little bit of training, not that much training, there's kind of a latent

(11:33):
reasoning capability that allows you to do inference time, scaling effectively.
'cause generally in my awareness, like you can do in first time scaling with basemodels, but it's not nearly as powerful as if you do additional training Yeah.
To make them capable of strong reasoning.
And what we've seen is with just a little bit more training,you go like, you know, 20%, 30% improvement from a base model.

(11:58):
So it'll be very interest to see if you can get thesame level of improvement with stuff like GB 4.5.
And then of course it has distillation question of, yeah.
You've seen, again, with base models, you can take a very smartmodel and kind of juice an intelligence into something smaller.
I could also see that being a reason to keep trainingbigger models and, and that is something that will change.

(12:21):
So, you know, like the cloud three opus for example,like nobody ever really used that because it's so big.
It's so bulky and expensive to query.
Right?
this is the problem with these super scaled base models.
When I say super scaled, I mean large parameter accounts, as you said.
In practice, what the labs do is they, yeah, they do distill these down to much, much smaller sizes.

(12:42):
So that it's cheaper at inference time.
That makes all the sense in the world.
You basically say, look, I'm not interested in making a you know, like a compute optimal model.
I'm not interested in, in making a model that is large enough for a given compute budget.
Like, as you increase your compute budget, you should beincreasing theoretically, the number of parameters in your model.
Those two things should scale at the same time.
That's what the scaling laws say, but increasing the number of parametersin your model means driving up the cost of inferencing that model.

(13:09):
And so what you'll often see is people will actually say, no,I will artificially keep my model smaller than it should be.
I will intentionally sacrifice performance because it's gonna cost lesson the backend if it's being queried by, Say billions of times a day.
So, and that, and that's actually gonna be another calculation when you do reasoning,because doing reasoning with these models means querying them many, many times.

(13:31):
So you're running them at inference time a lot more.
So the economics then, essentially what you're doing is you areliterally trading, trading for inference time compute, right.
Or you're, you're trading off your, your performance, let's say on in those two different regimes.
And so yeah.
Anyway, the lots to say there, but high level, I mean, I, I thinkyeah, betting against scaling at this point is, I, I wouldn't do it.

(13:53):
But, but I think I, I wouldn't bet against scaling.
But there is an interesting question as to traditionalscaling of increasing model weight counts at pre-training.
That's an interesting situation.
We are in I guess when I say scaling, I usually think of compute.
I think most people are thinking in that, that direction.
But you're right.
Like there's, there's those two things for sure.
Yeah.

(14:13):
Well, lots to say there, but we're gonna have to move on to the actual news, thenew things this week, starting with tools and apps and, and very much relevant.
Our first story is about QWQ 32 B. That's from Alibaba and that is their new model thatis as good as deep Seq R one and is seemingly o outperforming opening as O one mini.

(14:38):
So this is very similar to R one in a way.
They took their base model, QWQ Max, I would guess that was released about a month ago.
They did some additional training to make it a reasoning model withreinforcement learning, and then they found that it is able to do all of these
benchmarks at about the level of deep CK one, slightly better, slightly worse.

(15:02):
Pretty much equivalent and we don't know much.
We didn't release a paper, but in their brief blog posts, I dosay that this appears to be very similar in terms of approach.
They start to model, they train it on coding task and math tasks whereyou can have automated verifiers without any sort of trained reward model.

(15:23):
You just, you know, train specifically on coding and math because it'svery easy to evaluate your final answer with some hardcoded rules.
And you do a little bit of training beyond that as well, which I think is interesting.
R one had this two stage thing.
This also has this two stage thing where you start with just the pure codeof math and you do a little bit of broader training with reinforcement

(15:45):
learning on kind of general problems, and they use the trained reward model.
So, again, not too much beyond that we can say on this model, butI guess it again, highlights that if you have a good base model, it
honestly, it doesn't seem that hard to get to a good reasoning model.

(16:06):
And also, of course, this was kind of a big announcement from Alibaba.
They already have this up on their Coin Chat platform.
They're also releasing weights and their stock jumped, I, Iforget, like five, 10% a decent amount based on this news.
Yeah, I mean, it does seem to be a performance model.

(16:27):
It is now running, I guess what are we, it was like January 20, so about two months after deepseek, you know, all kinds of other things delay the, publication or the release of these models too.
So it's hard to know exactly when the training runs stop, butit, it does seem like they're, they're fairly close behind.
Yeah, a couple, a couple of notes.
I mean, they, they make a big deal out of the.

(16:47):
The number of parameters in this model relative to Deeps Seq and Deeps R one.
So this actually ties back to the conversation we were just having.
This is a 32 billion parameter monolithic transformer, whereasDeeps Seq was a, I dunno, 600 something billion parameter.
They had like 37 billion activated parameters.
So deeps R one was a mixture of experts, essentially, you know, query goesin and it gets, or a prompt goes in, it get, gets fed, or token by token

(17:14):
that is, it gets fed to a couple of specialized, kind of expert models.
And then there's, anyway, there's one expert I. That always gets, gets queried.
It, it's a whole thing.
Go check out our episode about that.
But bottom line is every time you, you inference deep seek R one the vast majority ofthe parameters in the model aren't actually involved in generating the final output.

(17:36):
And so there's only 37 billion activated parameters for each forward pass.
And that contrasts favorably to 32 billion, a smaller number for this model.
But in this model, every single parameter is activated every single time.
So this makes it hard to do an apples to apples comparison.
The other thing is.
It is much more convenient to have a smaller model.

(17:57):
32 billion parameters means you need a lot less ram to hold the model.
Just yeah, easier requirements, so that's a, an application level thing, right?
If you're a, a person, an engineer who wants to take this model and dosomething with it, then for sure you know, Quinn with questions 32 B
is gonna be a more interesting model potentially than Deeps Seq R one.
'cause it's just smaller kind of, you, you need less ram, less infrastructure.

(18:19):
But from a scientific standpoint, when we ask ourselves how how farbehind or ahead is Alibaba relative to Deeps seq, we actually can't.
Totally know until we know the compute budget that went into this.
See, it might be a smaller model, but it could justbe way more overtrained relative to deeps CQ R one.
And without knowing that, it's really difficult to know.

(18:40):
Alibaba certainly has a much larger compute fleet to throw at this kind of thing.
You know, deeps, SEEQ had, at least for, for R one, youknow, you're talking a couple thousands of, of GPUs.
So all in all like difficult to tell.
If you're curious about the competitive landscape of Alibabaversus deep seek, which one to pay more attention to?
There's not a ton to sink our teeth into here until we knowthose compute numbers, which may or may not be forthcoming.

(19:05):
One thing that we do know is, so they say we've integrated agent relatedcapabilities into the reasoning model, enabling it to think critically while
using tools and adapting its reasoning based on environmental feedback.
So there is presumably some kind of supervised, finetuning Would make the most sense here for tool use.
some SFT stage, it's not purely just pre-trained and then straight to rl, which was the case fordeeps seq R one zero, but not Deeps seq, R one, deeps seq R one did do a little bit of supervised

(19:32):
fine tuning where you explicitly train the model to use certain tools to behave in certain ways.
In the case of R one, I think it was more to reason in a, in a specific way or intelligible way.
So anyway it's an interesting model.
the benchmarks do say at least the ones they publish and they arecredible benchmarks that yeah, this is on par with, with R one.
Sometimes a little better, sometimes a little worse depending on the situation.

(19:54):
But Alibaba certainly has absorbed at a minimum many of the lessons from deepsseek and wouldn't be surprised to see them actually take the baton and, and scale
the blazes if deeps seek, can't get their hands on as much compute as Alibaba.
That kind of becomes a really interesting competitive differentiator.
Hmm.
And just to add a little bit to their overview to be precise.

(20:18):
So the timeline is they released Quinn 2.5 max in late January.
That was their, you know, frontline model.
They also, at the time released Quinn 2.5 dash 1 million.
That was the long reasoning.
Then just a couple of weeks ago they released QWQ Max coin of questions.

(20:42):
So that was also a reasoning model, and at the timethey released that to que chat and had the reasoning.
So, but essentially here with qw, Q 42 B is a, as you said, it's smaller.
QWQ Max was also in the MOE model built on QU 2.5 max.
This is, seemingly smaller.

(21:06):
And in a blog post I do highlight this one is the one that strained reinforcement learning.
So presumably Q wq Max was trained with supervised learning on tracesof reasoning from maybe other models versus this trained from a base
model just via a l and made to match the performance of other reasoners.

(21:29):
And onto next story, we have an actual product announcement, not just a model, and it is Alexa plus.
So Amazon has given a presentation, this is not out yet, but they did provide an overviewof the next iteration of Alexa that will integrate chat, GPT chat bot type intelligence.

(21:53):
So they will have many new features.
Obviously you can probably chat to it, but it can do various complex tasks like,I don't know why people love is booking a re restaurant and managing travel.
Examples like why would you want a chat bot to book you a flight?
But anyway, they did g give those examples.

(22:16):
It can remember things you tell it to remember and then it can do smart things based on that.
And they also releasing a revamped Alexa.
App and an alexa.com website that will kind of be bundled here and include multimodal capabilities.
So in addition if you have, for instance, an h created Amazon Camera Alexa will seeminglybe able to look at the video feed and answer questions about the data feed there.

(22:50):
So a bunch of new features integrated, and it does seem like this will bewith a subscription price of $20 a month to be able to use the Smarter Alexa.
Yeah, Amazon, you know, looking more and more like Apple.
In some ways with respect to this kind of scalinguniverse you know, they're not releasing their own models.

(23:12):
They're, they're focusing more on, on leveraging models builtby third parties and then integrating them into their hardware.
That actually sounds very, very much like the Apple play.
The difference being Amazon is much better positioned from a hardware standpoint with thetra chip than Apple is with their kind of weird fleet of of kind of more CPU oriented data

(23:35):
center infrastructure, which is kind of, is something we, we should talk about at some point.
But yeah, so this is Amazon basically saying, look, we are a platform for these models more.
So at some point they'll have their own, they have an a GI teaminternally that we've talked about before, but interesting to see them
kind of reaching for, for third party models, it makes sense, right?
The more you identify as a platform, the more you're servicing hardware rather than software.

(23:57):
You wanna commoditize the compliment, right?
That's the classic Microsoft play PCs are really cheap,but you know, what's expensive is the software, right?
So if you're in the business of building software, you wanna make PCs really, really cheap.
and make it easy for people to get to the point where they pay you for, for the,the good stuff here, the compliment to good hardware, it's kind of the opposite.
You know, if, if you're a hardware company, you wanna commoditize the LLMs, haveOpenAI, compete with Anthropic, compete with Google and then you get to have that

(24:24):
competition play out on your platform, drive prices down at the language modellevel, drive value up, and then your hardware suddenly becomes more valuable.
So I think that's kind of the play here.
And I'm, I'm curious to see too, from a model standpoint, you know, what is the, theadvantage going to be to Amazon to having, if they can unlock this, just more data
flowing through, can they leverage that into actually being competitive on the model side?

(24:47):
But for now, it's interesting.
Amazon's definitely struggled on the Alexa side.
Like we've heard a, like all the jokes.
it's definitely been a, a product in need of a revamp.
So maybe this will do it.
Exactly.
They did mention that this is built on the Amazon Nova model, but it can also leverage Enro models.

(25:08):
And they, I think it's interesting that they did also mention alexa.com as a newwebsite that seems to be basically like a chat GPT, like a chat bot interface.
You can upload documents, you can just have access to a chat bot.
And one thing also to mention is it will cost you a monthly fee unlessyou are an Amazon Prime subscriber, in which case it'll be free.

(25:33):
So in that way it's almost similar to X, where, right, if you'rein the Amazon ecosystem already, this might be your LLM of choice.
You would just use alexa.com because that's the thingthat you already have bundled with your subscription.
So also I think interesting from that aspect, maybe Amazon will have a chanceto actually compete in the just pure chatbot space in addition to its, you know,

(26:02):
smart hardware having more reason to be useful beyond what's been the case.
And moving on.
Next up, we have another demo of something that's upcoming.
The title of article is another Deep Seek Moment, question mark.
General AI Agent, Manus.

(26:23):
Manus shows ability to handle complex tasks.
So this was a bit of a web demo.
They had a released video on X and are now providing this as an invitation only web preview.
And this seems to be another really big investment in the agentic space.

(26:48):
So similar to something like Claude Code, you can task it with a really large task likedeveloping a website, developing, you know, an app, and it would go off and do this for a while.
It'll go and do it for, you know, minutes, potentially dozens of minutes.
It could accrue like, you know, $10 of inference cost, but thenin the end it could actually spit out a wholly working website.

(27:16):
So not too unique from what we're showing, but another indicator that this whole agentic directionthat, you know, has been a focus for much of 20, 24 is finally starting to sort of come together.
Yeah.
And another, you know, Chinese company moving this direction too.
So yeah, it'll, it'll be interesting to see.

(27:38):
Right now it's the, waterline is definitely rising on agents andI think the, the biggest question is gonna be, you know, whether.
Whether they can marshal enough compute to make these competitive with the Westernagents that basically have not unlimited Nvidia GPUs, but you know, closer to it.
And 'cause at a certain point it's a question of like, okay, you, you've shown that you can getlift off on inference time compute, but now you have to imagine, as we speak, every fleet of

(28:06):
GPUs in the West is being realigned around the thesis that inference time compute is a big deal.
And so rather than having, the say.
Oh three mini or, or oh three level inference, time budgets thrown at them.
Now we're gonna see really like, what does this look like when at industrial scale?
We're trying to trying to ramp that up the same way we did with pre-training.

(28:26):
So I think it's gonna be a couple months before we see that generation of, ofagents and then we'll be able to get a better sense of like, what does the long
term or medium term equilibrium look like between the US and China on this stuff.
But apparently, yeah, Manus does claim to outperform even opening eyes deep researchbased on on some, some benchmarks that are focused on general AI assistance.

(28:48):
So kind of interesting breakthrough.
Yeah, it's an interesting release too.
They got a lot of, let's say, hype or attention based on this post on X.This is a small company, it's Butterfly Effect, has a feud as an employees.
So it also varies some skepticism here as to how real this announcement is.

(29:09):
But regardless an indicator of where we are at kinda more broadly.
And next we have something from Microsoft.
It is Dragon Copilot, which is an AI assistant for healthcare.
So the main focus here is listening to clinical visits and creating notes.

(29:30):
Essentially it's like a AI that can be used for voice dictating and listening on a conversation.
Be an ambient listener.
Microsoft actually acquired a company called Nuance that was specialized in ambientlistening and voice dictating, and then it is able to create notes for doctors.

(29:53):
So we've seen this as an idea for quite a while.
We've covered, I think, multiple cases of this idea of AI asa note taker for doctors as being potentially a useful thing.
And another example here of, you know, now that Microsoft is providingit, presumably it's a pretty mature product offering of this kind.

(30:14):
Yeah, the, the challenge as ever is in medicine, doctorsare notori notoriously hesitant to use new technology.
and especially so I'll say this, a lot of my friends are doctors.
I've had these conversations with them.
Ego plays a huge, huge role in medicine, especially among doctors.
So you'll see rollouts of, of AI models that can say produce an end to diagnosis.

(30:36):
And there was a study recently I don't know if it might have beenduring a week, I was away, but it, it was one of these amusing things.
I, I had some, some fun conversations with some friends of mine in medicine.
So it turns out that if you have a, doctor working with and I forget what model they were testing.
I think it might have been four.
Oh.
But if, if you have fine tuned four oh or fine tuned, you know, approximately Frontiermodel, doctor Alone will underperform the model and also underperforms Doctor plus model.

(31:06):
That's actually really interesting.
So the doctor with the model does worse than the model alone.
Literally this is the, the doctor basically taking on average correct answers that the model givesit, and it gives them, and then, being like, no, no, that can't be right, or something like that.
That I'm obviously caricaturing a little bit here for humor, but somethinglike that is, is happening under the hood in a lot, in a lot of these cases.

(31:27):
It's part of the psychology of the space.
So really, really challenging and this is why you're seeingapplications like this that are like, you know, not opinionated.
I'm just taking notes.
I'm just being, you know, a good little bot sittingin the background, not telling you how to do your job.
for cultural reasons, I think that might be pretty good use case to go after at this point.
Right.
I do think so.
We've seen some research that indicates that this kind of technology helps reduceclinical burnout helps have patients have better experiences by a pretty wide margin.

(31:57):
And this has been being tested in, nine hospitals and someother clinical locations partnering with well Spend Health.
So it seems that they are already testing us and it'llbe generally released in the US and Canada in May.
So I guess it'll be interesting to see if now it willstart going to doctors if they'll have a recorder going on.

(32:23):
I don't know.
And onto the last thing, we have a new offering from Misal and it is their O-C-R-A-P-I.
So OCR is optical character recognition.
It's basically looking at a photo or a PDF and converting that into the actualtext contained within that photo or, you know, scan of a page or whatever.

(32:47):
And we've seen leaps and bounds improvements in o osteo technology over recent years.
This offering by mis mistral is basically, you know a version of that that you can use via NAPI.
And they say that this can play into allowing LMS to use and, and talk about PDS andscans of things that perhaps we're not good at with just pure multimodal reasoning.

(33:19):
And out to applications and business.
And we haven't talked about philanthropic yet, so I guess it's their turn.
And we haven't talked about anyone getting billions of dollars in a little while.
So that is now also what's happening.
Anthropic, which we've covered, has been in the process of fundraising over the lastcouple months, has now completed that fi fundraising round and is valued at 61.5 billion.

(33:48):
That's rising from 16 billion.
That was just over a year ago.
They got 3.5 billion in this round.
It was led by Lightspeed Venture Partners and it, makes it that philanthropic hasnow raised over 14.8 billion, so nowhere near OpenAI as far as I can remember.

(34:14):
OpenAI, I, I lost count of the number of billions they have gotten and burn through.
But another indicator that I think philanthropic is.
Still the front runner competitor to OpenAI in this space?
Yeah, for sure.
I think the, the latest is from OpenAI.
They're in talks to raise at a $300 billion valuation remains to be seen,whether that materializes obviously, but it's also pretty consistent.

(34:38):
You know, they've, they've been about a you know, four orfive x multiple ahead of anthropic for the last little bit.
So this suggests, you know, consistent growth curve for both companies.
Maybe not too, too surprising.
A bunch of new investors who are joining the Round, it istheir series E general Catalyst, Jane Strait, fidelity.
I. Okay, Menlo, were already pre-existing investors as well as Bessemer.

(34:58):
So, I mean, you're really, really high quality VCs.
No surprise there either.
their claim is they were originally planning to raise $2 billionbut ended up with an oversubscribed round, so, could be true.
It's also sometimes a trick that you'll see used where people will say, Hey, we'reraising this amount of money, and the amount that you give is like, less than what
you're actually trying to raise, just so that it creates fomo and you get more investors.

(35:19):
But yeah, in any case, that's a, a really big raise.
They're one of the, the bigger private companies in the United States right now.
You think about, you know, $300 billion valuations like OpenAI and, and SpaceX.
You know, there, there aren't that many companiesin that range including at the 60 billion threshold.
So.
Pretty, pretty interesting.

(35:40):
Yeah, we'll see.
Apparently, no surprise anytime you see this.
They're using the capital to develop the next generationof their AI systems and expand their computing capacity.
So there you go.
more GPUs.
And next up we have an IPO Story and video backed Core Weave hasfiled for IPO and has also reported a revenue of $1.9 billion in 2024.

(36:05):
Core Weave is a cloud computing provider that's backed by Nvidia and they areaiming to raise $4 billion through the IPO aiming at a valuation of 35 billion.
So a player here in the cloud space, they have been around since 2017, we're into crypto mining.

(36:27):
Guess now they're aiming to be part of the infrastructure play for ai.
Yeah, of the virtues of going for an IPO is you end up, you know, publishingall of your your self-assessed weaknesses and, and differentiators and all that.
So we get a little bit more of a sense of where they're at.
Turns out about 77% of their revenue came from just theirtop two customers for 2024 one of which was Microsoft.

(36:51):
So and Microsoft, by the way, accounted for apparently two thirds of overall sales.
It's a pretty lopsided situation that that is, you know, some structural risk.
And we've seen similar things with with other sort of cloud companies like this.
It's just the nature of the game.
And yeah, I mean, pr prior to this, they had rated the $23 billion valuation.
So, this is about where you'd expect them to be is, going foran IPO after that level of scale to, to raise more capital.

(37:18):
So yeah.
Just as a, a reminder, like core weave is, I. A pretty interesting company.
We've covered them quite a bit.
They have an NVIDIA partnership.
NVIDIA is one of their, their most important investors that you can imaginehelping them get access to GPUs a lot faster, which is a big differentiator.
And then they also are known for this very flexible pricing model where they give youjust a lot more like granular and cost effective pricing for a lot of GPU resources.

(37:46):
So like you can rent individual GPUs instead of entire clusters, for instance.
So it's a lot more, a lot more balanced and easier theoretically for smaller players to use as well.
So, sort of interesting expect to hear more about Core Weave in the future, that's for sure.
It's expected by the way to trade under the nasdaq.
So that's that's gonna be their, exchange of choice.
I. next.
We have Waymo and the news that their partnership with Uber has now officiallybegun the, access to Waymo vehicles through Uber in Austin has launched.

(38:19):
So now when you hail, do people say taxi these days?
I don't know.
An Uber vehicle through UberX, Uber Green, any of these?
You might get a Waymo, which is kind of interesting.
So you, you don't have any social specialized requests.
You just can't get matched to one.

(38:39):
You can adjust your rider preferences to increase your Chances of getting a Waymo.
And apparently pricing will be the same.
Just about for tip.
And this will cover 37 square miles in Austin.
And moving right along.
The next story is about Microsoft and open ai.

(39:00):
We covered quite a while ago the UK Competition and Markets authority.
The CMA had launched an investigation into the partnership between Microsoft andOpen ai due to some antitrust concerns, which was quite trend last year while
that investigation has concluded and they decided that the partnership is fine.

(39:24):
Basically Microsoft has influence over OpenAI but not control,and so it doesn't meet your criteria for a merger review.
Yeah, the triggering event here was just seeing how effectivelyMicrosoft was able to pressure the OpenAI board into rehiring Sam.
And so anytime you see that, it raises questions about,okay, well, does Microsoft then have effective control?

(39:49):
And that would trigger antitrust concerns.
As you said, they instead found a high level of material influence, notoutright control, not something that would justify further action here.
they are saying quote, the CMAs findings on jurisdiction should not be read as thepartnership being given a clean bill of health on potential competition concerns.

(40:09):
But the UK merger control regime must, of course, operate within the remit set down by parliaments.
Basically, they're arguing that just within the narrow regime that they arecharged with, this does not qualify as something that they need to act on there.
There has been criticism of this, it's interesting to note the laborgovernment under kiir starmer that's just come in is much more.

(40:30):
Sort of pro AI acceleration economic growth type stuff than the Rishi Sunna conservativegovernment that just just left, that kicked off, of course, the famous AI Safety Summit series.
And so instead what you're seeing is appointments within the government that reflect that view.
And so there's concern.
For example, there's one quote from somebody who said the CMA has sat on thisdecision for over a year, yet within just a few weeks of a former Amazon boss being

(40:57):
installed as its chair, it is decided everything was absolutely fine all along.
Nothing to see here.
Hard to really know how far all this goes.
Because obviously the, the machinations that go oninside a a government body like this are quite complex.
Partly political, but also just partly this could have just happened this way.
Really difficult to know, but that is part of the conversation happening around here.

(41:18):
And the last story here is about scale AI announcing a multimillion dollar defense deal.
So they have a deal with Department of Defense in the us It's called the Funder ForgeAI Agent Program, and it is meant to enhance US military planning and operations scale.

(41:38):
AI is sort of the leader.
On this initiative and said they LB partnering or, or usingtechnology from Microsoft and, and dural among others.
And they do say this will be under human oversight.
But the idea seems to be to add a genetic capabilities.
And you know, this is just another indicator of theoverall trend of a movement of tech towards military.

(42:06):
You know, friendship with OpenAI and Nero and others haven'tmade this similar kind of maneuvering over the past few months.
There's not a ton of like I would say concrete, non-obvious information that's out there.
Right.
They're calling it a multi-million dollar deal.
Well, no shit.
You're talking, okay.
we know it's spearheaded by the DIU, the Defense Innovation Unit.

(42:28):
which is Yeah.
Roughly what it sounds like they do a lot of advanced r and d for Department of Defense.
By the way, I mean, there's been a lot of concern around a. The obviouslyuse of like fully autonomous systems to do targeting and things like that.
The claim is that that will not be, there will be a human in the loop on some level.
but in any case, the reality of, of the spaces we are heading for a world where USadversaries absolutely will be deploying these systems without any kind of oversight.

(42:54):
And at a certain point, the response times of thesesystems is just too fast to have a human in the loop.
So anchoring on, you know, whatever human in the loop means, 'causeit's already kind of ambiguous is don't think a winning strategy.
I think ultimately the economics, the geostrategic landscape forcesyou in the direction of full automation, whether you like it or not.

(43:15):
And chalk it off to a, a, moic problem, whatever you wanna call it.
But they are saying here that they're partnering with Enduro, which is, you know, thefamous Palmer Lucky company that's, like you know, anyway, on its way to becoming a big
defense prime, basically and Microsoft, it's gonna be about AI agents, no surprise there.
And the use cases they cite here are modeling and simulation decisionmaking support, proposed courses of action, and even automated workflows.

(43:40):
The rollout, by the way, is gonna start with US Indo Paycom and US ucom.
Those are so the US DOD is set up with different, what are known as, as combatant commands.
These are the essentially integrated.
Operations that actually do real shit.
So, you know, in, in the Middle East you have us centcom, US Central Command.
you know, they're in charge of, whatever happens in Syria, for example.

(44:02):
Interesting that it's being rolled out in, in, you know, IndoPaycom, so Indo-Pacific commands, so that, that includes China.
and then us ucomm presumably with, Russia and the Orbit, these are interestingtheaters to be experimenting with some of these things because you get actual
sort of like rubber meets the road impact pretty quickly, I would imagine.
So.
Interesting.
And Alex Wang, by the way, CEO of scale ai actually pretty concerned about alignment issues.

(44:27):
So he's, he's doing this with that in mind.
Very thoughtful guy and we'll, we'll talk about something that heput out with Elon's AI advisor, Dan Hendricks recently as well.
But anyway, he's, he's on the map with this and scale.
AI is obviously doing a lot more DOD stuff these days.
And moving on to projects and open source, we beginwith deep seek that had a whole open source week.

(44:50):
So just to quick give a quick overview, they had aweek where every day they released a new repository.
So the whole list is flash, MLA, an efficient MLA decoding kernel for hopper GPUs, deep ep,a communication library for mixtures of experts model, deep gem optimized general matrix

(45:12):
modification, library optimized parallelism strategies, a framework for optimizing app parallelismand fire flyer file system, a whole file system optimized for machine learning workflows.
And finally, deep seek V three R one inference system.
So that's.
Six different packages that they released.

(45:34):
And as you can tell from that overview, it's very much focused on infrastructure.
This is part of their secret sauce.
Part of why deep Seek V three was so performant at sucha low cost, it just optimized the crap out of everything.
Like working into their own matrix, multiplication, cores, and so on.

(45:56):
So they have now shared that all that world really exciting.
I, I I suppose for some people work on this sort of stuff.
Gi me I'm sure you're able to provide more detail here.
Yeah, I mean, well, so this is in a way the big reveal of all the stuffthat went into the V three and R one paper and, and some other stuff too.

(46:16):
I think part of what it shows you is just the incrediblebreadth of engineering talent that they have, right?
So in some ways the Firefly file system, the FFF.
S is the least kind of compelling, least to me in terms of what it actually gets you, butit just helps to illustrate how frigging wild and and far their, capabilities stretch.
So you know, this is basically for, SSD storage.

(46:39):
So, there's, there's this issue of, of what's called the read throughput.
It's the rate at which data can be read out of storage, like long-term storage in your data center.
And by long, long-term we really mean storage here that has a really high capacitythinking, like storing model checkpoints or, big chunks of data sets and things like that.
You don't use them very often but when you do, you need likereally, really large throughput you're pulling off tons of, data.

(47:03):
So, looking at terabytes per second, usually in that scale.
So.
It turns out that they've been able to achieve 7.3 terabytes per secondof data kind of read throughput from, from their new optimized setup.
That's actually really, really impressive.
It's up there like frontier scale training, runinfrastructure, like, but now it's being open sourced.

(47:24):
Anyway, there, there, there's a bunch of detail in here about how it's optimized.
I'm actually gonna skip over that completely, but we can dive into it.
It's just like, this is, even at the SSD level, they're like optimizing that.
Even though SSD is not a key bottleneck, typically for like high performancecomputing, AI runs like high bandwidth memory is GPUs are like flops or network

(47:47):
interconnect, but it's not usually pulling stuff off of long-term storage.
I think the, the big story is really more the dualpipe release or, anyway, that's one of the bigger ones.
Which, which we can talk about as well.
That one.
Yeah, I think that one, if I'm not wrong.
So they have another release that wasn't even part of thisweek of open source that focused entirely on infrastructure.

(48:12):
This was part of the overall, week.
This is day four of the week.
So this is interesting.
I think and you'll go into depth here, is because this ismore of an algorithm, so it's less just pure infrastructure.
This is a new kind of inference technique.

(48:32):
And then I'll let you take over on, yeah, I mean, it's, it's so hard to tell what, whatqualifies as hardware and what qualifies as a software at this point, but you're right.
I mean, you, you could argue it many different ways, I guess.
First we gotta talk about this idea of pipelineparallelism, which is a, a, a way of breaking up your.
Your training task that basically everyone does when they doreally, really large scale training runs deep seek certainly did.

(48:58):
Essentially you can think of this as having your models layers dividedsuch that, you know, maybe layers one through three are sitting on
GPU one layers four through six or sitting on GP U2 and so on, right?
So different layers of your model are sitting on different GPUs.
Now, typically when you do this you have to feed.

(49:19):
So let's say you get a new piece of data you wanna feed toyour model, well, now you gotta feed it to GPU number one.
Have it process that, that mini batch, then that output has to be fed to the next GPU.
You gotta move it over to the next GPU and that next GPU then startscrunching on it while GPU one starts processing the next mini batch.

(49:39):
Now, one thing you might notice about that setup is it means that you've got awhole bunch of GPUs that are holding the weights for the later layers in your model.
Just kind of sitting idle at the beginning, waiting for something to do because you have tosend your, your data, you know, through the first few layers before they get to the last ones.
And so that creates this effect that's known as a bubble, right?

(49:59):
You have this, this bubble that forms during pipeline parallelism.
And, and it's like you got all this idle, you know, GPUs that aren't being used.
What deep Seek does is they're trying to minimize that one.
One thing they're doing here is they're trying to minimize the size of that bubble.
And one technique they use is having a forward pass.

(50:19):
Start with the.
GPU that's holding the earlier weights, the, the earlier layers of the model.
So forward pass there while a backward pass starts to go through the, the end of the model.
And data propagates towards the middle.
There's this meet in the middle problem that has to be resolvedand they find, anyway, really interesting ways to solve for this.
One of the key strategies that they use is finding reallyclever ways to overlap communication and computation.

(50:44):
And, without going into too much detail, I do wanna talk about.
The, this thing called the streaming multiprocessor.
So your GPU, you can think of like the workhorse unit thatdoes work on your GPU as the streaming multiprocessor.
And there are tons and tons and tons of these on A GPU.
They can handle like.
Computation.

(51:04):
So they execute code whether that code is for, for actually doing like mat malls and, andother forms of compute or for communication, but they can't do both at the same time step.
So a given streaming multiprocessor is either packaging data to send it you know, onthe network to other GPUs or somewhere else, or it's performing an actual computation.

(51:27):
And what makes the deeps approach so sophisticated?
That they've really carefully allocated some streaming multiprocessors to handlecommunication and others to handle computation on the same d on the same GPU.
And so you imagine like at, at a given time step like time, step one streaming multiprocessorslike one through 80 are doing, you know, a computation on batch a, whereas, you know 81 through

(51:53):
a hundred are handling communication for some previous set of computations that they ran.
And so you've got essentially one single GPU dye, whichat the same time is actually doing many different things.
And this helps you avoid this meat in the middle problem that comes fromfeeding data you know, at the lower layers and the top layers at the same time.

(52:13):
'cause now you've got these like kind of schizophrenic GPUs that are.
Having to do multiple different things with data flowing essentiallythrough one process in one direction and the other in in another.
it's actually really, really fascinating.
They also chop up their layers in a very creative way separating out the multi-layer perceptronsfrom the attention from all this stuff allowing them to have more fine-grained control.

(52:35):
And what they claim is that they basically get perfect communication, computation, overlap.
So you don't run into this problem where you have idlecores waiting to be fed data that they can crunch.
There's always something going on in the system.
So anyway, that, that's really the dual pipe advantage here and called dual pipe.
'cause you're feeding both in the, you know, early layers, you're feeding inthe forward pass and then the later layers you're feeding the backward pass.

(53:00):
Very cool and moving right along.
And next up we have a release of a model and it's from physical intelligencewhere open sourcing, where PI is zero Robotics foundation model.
So we covered this a few months ago.
PI is zero is a model that takes a video stream, takesa task specification and outputs robot control for.

(53:21):
A variety of types of robots.
Now, VA PI zero model and its code are available on GitHub in a repository.
And I think it's also getting integrated into hugging face their robotics framework.
They also have a few variations.
They have a PI zero fast base model, and then they also have some other oneslike PI Fast droid that's fine tuned for a franca robotic arm in particular.

(53:51):
So one, one of the first releases of a large foundation modelphysical intelligence got like $70 million in funding as a startup.
So they have had the resources to collect the data sets and train the models that.
Have so far really not been, kind of possible.

(54:12):
And so they, they do say that you can fine tune it to your own taskor applications with around a few hours, one to 20 hours of data.
last story.
On the open business front, we have a big bench, extra hard from DeepMind.
So as we've said, for benchmark, increasingly not useful.

(54:33):
And this is yet another demonstration of that big bench.
Extra hard is, building upon big bench hard.
It replaces its 23 tasks with more difficult counterparts that require advanced reasoning skills.
And as a result the state of art LLMs reach a top accuracy of 20 feet 0.9 for adjusted base models.

(54:57):
And they do reach 54.2% pass rates on the reasoning specialized models.
So already, you know, sort of able to do these tasks, but clearly a lot of room to improve.
Yeah, I, I remember back when a new super challengingbenchmark, you know, you'd have like 1% 3% performance on it.

(55:17):
And now we're starting this off with a view to making it the really hard benchmark.
And it's like, we're already at 45% basically.
So I will say that that is O three mini high like on on high compute mode.
and a lot of the reasoning models like deep Seek R one and Stel R one, Quin 32 B,they, they're hitting, you know, sub 10%, you know, like around 5% performance.

(55:38):
So a lot of room to improve there, but certainly you can see a big step change, right?
O three mini high.
One of these models is not like the others.
And and that's quite interesting.
I think it tells you something about the anyway, theoptimization process that they're running in the back end there.
So very interesting.
Another, instance of benchmark treadmilling.

(55:59):
I think we're gonna find, like, this one's gonna get oldreally quick especially in the world of inference time compute.
'cause, performance on benchmarks is just, it, it's moving a lot faster than it used to.
And speaking of improving reasoning models, we are moving on to research and advancements,and the first paper is cognitive behaviors that enable self-improving reasoners.

(56:22):
So this paper is asking a question of how do reasoning models actually do reasoning?
What kind of patterns of cognitive behavior lead to effectivereasoning and they identify four specific behaviors.
Verification meaning that you verify your solution, backtracking, going backand revisiting previous decisions, sub goal setting and backward chaining.

(56:52):
And if you've used reasoning models, I think intuitively it, this makes sense.
This is kind of what you see them doing often is listing out their steps.
You know, think, step by step is the classic, you know, quasi reasoning thing atthis point, but it used to be how you get it to be better at more complex tasks.
And so they, suggested specific inference techniques and show thatif you train for them in particular, you are able to do a lot better.

(57:22):
Yeah.
And the kind of archetypal examples that they use inthis paper are so Quinn 2.53 B and Lama, 3.23 B, right?
So these are essentially roughly the same.
I mean, they're the same scale, 3 billion parameters and roughly the same generation.
So Quinn and, and what they find is.
If you use the exact same RL reasoning training process,they, the Quinn model will vastly outperform the llama model.

(57:49):
And so this gives kind of the, the initial prompt, right, that has them asking, you know,what are the intrinsic properties that enable effective self-improvement at the RL stage?
The toy environment they're gonna use for this, by the way, is a game called Countdown.
So you basically have a set of numbers.
So, imagine I give you like a bunch of different numbersand then you have to use four basic arithmetic operations.

(58:12):
So, addition, subtraction, multiplication, divisionto, to combine those numbers to get a target number.
So, roughly like some kind of Sudoku ish thing.
and it's on that, that they're gonna do their RL optimization andthen they're gonna compare the performance of different models.
And so what they find is that the Quinn model naturally shows reasoningbehaviors like verification, like backtracking whereas Lama initially

(58:34):
lacks them just the base model right before any kind of RL loop.
And so, but what they find is if you prime llama.
With examples that contain those kinds of reasoning behaviors.
And especially backtracking, they see a si like a significant improvement in itsperformance during RL performance actually goes up to the performance of Quinn.

(58:55):
And so that's kind of interesting.
It's also interesting that if you prime those models by essentially givingthem the right reasoning process or examples of that kinda reasoning,
but incorrect solutions, you still get identical performance increases.
So it's almost as if the reasoning process is the whole thing and the,final solution in, in the training context isn't even that important.

(59:19):
Now, I do think that it's important to kind of.
Contrast that with what we learned in the deeps seeq R one paper,because on this, the face of it, this might sound kind of contradictory.
So if you recall Deeps Seeq R one, right?
They take their base model and then they do reinforcement learningand all they care about really is just did you get the right answer?
And the reinforcement learning process ends up causing the model to learnthese reasoning behaviors just by forcing it to get the right answer.

(59:46):
And so this might seem like a contradiction because what this paper is saying isactually if we make the model reason using the right reasoning of strategies even
if our training set contains incorrect final answers, it'll end up performing well.
the distinction I think here is well you are priming them with incorrect solutions,but you're still then also training it with a l that does provide the right solution.

(01:00:10):
Yes.
So they're, you know, training to get the right results, but primed with this is howyou should reason, and then the actual kind of final step of reasoning is not Right.
Exactly.
Right.
So you're giving, like in context, you're telling it like, Hey, use these strategies,and then suddenly the performance goes up even if your answer in context was wrong.

(01:00:33):
Sorry, if, if I, if I said that if I said trained rather than primed at some point, I apologize.
Yeah, exactly.
And so this is also suggests that there's like a latent abilityin the llama series of models to use this reasoning capability.
Right.
The base model has this like latent reasoning ability and that has this reinterpretI. that we haven't had cause to do this before, but it has us reinterpret the RL

(01:00:58):
stage as more of a capability elicitation process than a capability creation process.
Essentially finding ways to mine the, the reasoning abilities of the base model.
So yeah, I thought it was super interesting.
And and one useful arrow in the quiver if we're looking to get moreperformance without necessarily spending a ton on compute in the RL phase.

(01:01:22):
Just like better prompts, right?
That actually explicitly show these, these reasoning strategies being used.
So kind of interesting, right?
And, and that builds on recent paper titled LLMs can easily learn toreason through demonstrations, structure not content is what matters.
I'm not sure if you covered that, but basically same ideastructure of how you do your reasoning is the important bit.

(01:01:46):
And then you can train efficiently if you take that into account.
Next up we have the mask, benchmark disentangling honesty from accuracy in AI system.
So as you might expect from that title, the idea is how do you actuallyevaluate honesty if VLLM might be accidentally getting something
wrong, as opposed to, you know, intentionally being dishonest.

(01:02:11):
Well, this has a novel evaluation pipeline that, looks at the underlying beliefof the model and then see if a LM says something that contradicts that belief.
And we have large dataset, 1500 examples, and they can be used to evaluatevarious lms and have shown that state of art LMS often lie when pressured.

(01:02:39):
Yeah, essentially they have a bunch of prompts that are designedto exert pressure on the model to give a certain incorrect answer.
And then they have a bunch of prompts that are moreneutral the like, so-called belief elicitation prompts.
And they basically just contrast the outputs of those two to assess whenthe model is being, accurate and honest or inaccurate and honest versus

(01:03:02):
inaccurate and dishonest, all the, the kind of different possibilities.
They also use.
So this is from Dan Hendricks, by the way, from the Center for AI Safety,who's done a lot of the early work on this field of representation engineering.
And so they, they use representation engineering techniquesto try to modify the model to make it more honest.

(01:03:23):
And it is kind of an interesting experiment.
So they basically, like they, they add a, a pretty simple developersystem prompt before the user system prompt that tells the model.
To be honest, that's one easy intervention that they try that actually has a prettybig impact, about 13% improvement in honesty or 11 to 13% depending on the model.

(01:03:44):
But then, then they try this, this technique that's based on usingessentially like a, a Laura, Laura strategy an adapter model.
Or adapter layers stacked on top of their model, which theyactually train to modify the representation of, of a given layer.
So what they do is they modify each input in the training set by adding a bitof text that encourages the model to be either honest or dishonest, right?

(01:04:07):
So you can imagine a yeah, like a, a a little bit of text that says, like,you know, lie in your answer to, to be really crude or like, be honest.
And then they look at the activations they get from each of those cases,the kind of truthfully prompted model versus the dishonestly prompted model.
And then they take the difference between those activations.

(01:04:31):
They get a contrast vector out of that, A vector that basically tells them what is the differencebetween the activations when the model's being honest versus when it's being dishonest.
anyway, they add that contrast vector to their actual representations during trainingand try to train the model to kind of move in the direction of honesty in that way.
It's something that they've tried sort of sim or that Dan's teamshave tried similarly in other contexts that work quite well.

(01:04:56):
And that, that turns out to work comparably well to the developer system, prompt strategy.
So interesting both a diagnostic and a bit of a treatment.
Obviously not perfect.
But one of the really interesting things too, that they highlight is that larger models.
Are actually like more scaled models are often more accurate, but not more honest.

(01:05:17):
They will often be kind of more right on accuracy benchmarks.
But they have a tendency to behave dishonestly more than, smaller models.
So it's kind of a, an interesting trend and pretty strong correlations in both cases.
And that's it for papers in a section.
We have just a couple more stories related to research.

(01:05:38):
The next one is about the some pioneers of Reinforclearning, specifically Andrew Bartow and Rich Sutton.
Winning the Turning Award.
Returning Award is a very prestigious award in the field of computer science andthey are winning it for their decades of contributions to reinforcement learning.

(01:06:02):
So not much more to say there.
They are quite famous figures in the world of ai and I think this is perhaps not surprising.
Certainly.
You know, a reasonable award to be given.
Yeah, I, I'm surprised it did take this long.
Like, you know, RL has been very useful in admittedly more niche domains,for some time you know, multi-arm bandit problems and things like that.

(01:06:28):
But, yeah, I, I can't help but think that the reasoning model wave has hadsomething to do with this, where suddenly, you know, we have good enough
base models that RL is, is the way, and it didn't happen with RLHF too.
That's another interesting thing probably because of all thecontroversy around whether RLHF is for true RL or needs to be rl.

(01:06:48):
But now we definitely have those use cases where it's like, yep, youknow, rich Sutton and, and co like definitely are, are coming in handy.
And the last story here is about opening ai, launching a$50 million grant program to help fund academic research.
So that's pretty much it.
It's meant to support AI assisted research through a new consortium called Next Gen ai.

(01:07:14):
Founded with academic partners like Harvard, MIT, and so on.
It will provide research, grants, compute funding, and API access.
Moving right along to policy and safety.
First up, we have kind of an opinion piece or, or discussion piecewith nuclear level risk of a super intelligent AI as Jeremy Previewed.

(01:07:37):
This is co-written by Dan Hendricks, an advisor to Elon Musk onsafety and generally an influential figure in the safety space.
Also covered by Eric Schmidt, interestingly, who if I remembercorrectly, is an influential figure who worked at Microsoft.
So basically it is making a point that, there is a comparison to be madebetween the current ai arms race or race between us and China, which I

(01:08:05):
think is kind of heating up as a concept ever since Deep CCAR one came out.
And so the whole piece is discussing how that could be compared to, nuclearweapons in the sense that super intelligent AI systems could be as dangerous.
Yeah.
So Eric Schmidt, by the way, is formerly the co-founder and CEO of Google.

(01:08:28):
But, but yeah.
Google.
Oh, my be, no, it's just another hyperscaler scaler these days, right?
But yeah, no.
So what's, what's interesting about this is it's, it's basically arguing take a step back.
The reason that we didn't end up vaporized during theCold War was that the US had nukes, Russia had nukes.
Everybody knew that launching one nuke would meanlaunching them all, and everybody would die, right?

(01:08:52):
So there's, there's no winners in, in that situation.
The idea that.
by kicking off a nuclear exchange, you would also be signing your own death warrant wasknown as Mutually Assure Destruction or Mad, mad Doctrine as it sometimes referred to.
it's like the key piece of of sort of geostrategic doctrine that takes us through the Cold War.
Some, some of argued in one piece, though there were close calls and it's not, notobvious that the Mad Doctrine ends up looking as clever as, as it it did in this universe.

(01:09:21):
In most cases, we may have gotten lucky and there havebeen, anyway, a lot of close calls and a lot of accidents.
So here the question is, is there a similar set of incentives, similarsort of game theoretic landscape that applies to super intelligent ai?
If you assume that in particular super intelligent AI is AWMD, which I, I think is very clearly going to be the case.

(01:09:42):
So if that's the case, then suddenly you have this question of like, youknow, China's not gonna allow the United States to, if they possibly can.
They're not gonna allow the United States to build super intelligent AI systems here firstbecause that would lead to a decisive and permanent strategic advantage to the United States.
There is no more China Chinese hegemony in that world.

(01:10:02):
Vice versa, the United States would find it completelyunacceptable for China to build these systems first.
And so now you have the makings of Incentives for intervention on the part of both countriesand other countries that don't have a prospect realistically of building super intelligence.
And so, you know, they're incentivized to try to take out each other's trainingruns, knock out each other's data centers with cruise missiles or whatever.

(01:10:24):
and maybe just maybe the authors here are roughly arguing.
This means that you hit this equilibrium where just as China's about to make an SI, or justas America is about to make an a SI, the other country knocks them out just on the way there.
So we never quite get there.
This leads to they hope some kind of stable equilibrium where youhave, you know, data centers built in rural areas where it's okay if

(01:10:49):
somebody takes 'em out the casualties and damage is relatively limited.
That's one of the recommendations.
Anyway, there's a bunch more recommendations around non-proliferationyou know, better chip export controls and so on and so forth.
It is a deterrence framework.
interesting argument as to how realistic this is.
This intersects with a lot of the work that, that we've been doing.
I, I will say it does not reflect our, perspectiveon what an actual stable equilibrium will look like.

(01:11:13):
It depends, for example, critically and entirely on the US and Chinahaving almost perfect visibility into each other's AI development
programs, which certainly is not the case for the US vis-a-vis China today.
Is plausibly the case for the China vis-a-vis the US 'cause our security is garbage.
But yeah, there, there were a lot of.
A lot of caveats to this.

(01:11:35):
I think it's, it's great that they put this out and I'm, I'm really glad that they did.
but it's gonna be part of the moving discussion on this field.
It's, it definitely is part of the incentives that apply to this space.
It's just not clear that it translates in the way that,they've sketched out here into real world con consequences.
I, I think that the equilibrium actually ends up beingquite a bit more unstable than, than maybe they, suggest.
So great to have this out there.

(01:11:56):
This will be part of the conversation.
Very high, high quality report.
I think it just needs you know, there, there's room for input here from the actual specialoperations and intel guys who, who deal with our adversaries on the front lines every day and know
what the capability services actually are and, and what adversary behaviors actually look like.
I think that's a weakness right now in the, in the report.

(01:12:16):
But but still a great contribution and, and yeah, there'sgonna be a lot of discussion about this going forward.
Yeah.
Yeah.
This, this op-ed on time is, in relation to a full report, as yousaid, titled Super Intelligent Strategy by Dan Hendricks, Eric

(01:12:36):
Schmidt and Alexandra Wang from Scale ai, which I think you mentioned.
And the full report is 32 pages, so, you know, fairly detailed, but also I thinkinteresting that, you know, the three of them, one safety guy and two kind of
business people wrote a whole thing on national, oh, Alex Wang is alignment pilled.

(01:12:58):
So yeah.
I mean, oh, okay.
He is safety.
Safety, but he's not kind of in the safety research space.
Totally.
Yeah.
Yeah.
And, and I think this is, this is one of the, the big issues.
This is why we work where we do Right?
With sort of intelligence, anyway, you know, PE-people outside the safety community.
'cause frankly, I think one of the big problems with the safety communityis they have an unrealistic perception of what is actually on the table

(01:13:20):
in terms of what our adversaries can do and, and would do and so on.
And so I think, I think this is where, you know, similarly, it's great to have Alex and, andEric coming in and with their backgrounds that are different and, and weighing in on this.
So yeah, absolutely.
You're right.
It is kind of diverse here.
Yeah.
And next up we have a story about GT 4.5 Meter is anorganization that deals with model evolution and threat research.

(01:13:43):
And this, that is actually what it stands for, by the way.
Model evaluation and threat research.
There you go.
I got it.
They got to evaluate GP 4.5 pre-deployment and released evaluations as,as they say, so they, in the support, you know, said that GP 4.5 is.

(01:14:06):
Unlikely to pose a large risk compared to non 4.5 models.
You know exactly what you would hope for.
And also, I believe what OpenAI covered in their system card.
Yeah, it's, it's a sort of interesting, I. Post because they're taking a bit of arisk with respect to their relationship with OpenAI to say some of these things.

(01:14:28):
So Meter is a company that gets contracted by OpenAI and Anthropic and other, other companies tolook, especially at their, yeah, the model, like self replication, self exfiltration type risks.
so they're telling us they, they got early access to a, a checkpoint ofGPT-4 0.5, not necessarily the final version, which has been a consistent
issue where OpenAI doesn't give them access to the final version.

(01:14:50):
And we're also being told that they were given this access a week prior to release.
So again as has happened before, a, a pretty rushed evaluation timeline.
They measured the model's performance on their general autonomy suite and, re bench, which is thisbenchmark that they made to measure how close to a, top line AI researcher their models are and.

(01:15:13):
as you said, you know, no elevated risk here.
They find that roughly speaking, G PT 4.5 is about as like, has a 50% chanceof succeeding at tasks at AI research tasks that take about 30 minutes.
So, they can do the work roughly speaking of an AI researcheras long as that work doesn't take more than 30 minutes.

(01:15:34):
That's kind of an interesting benchmark.
But they do make a point of saying this.
In the future, we're excited about collaborating with Frontier AI developers to investigatethe safety of their models over the course of development, not just in pre-deployment sprints.
This comes up a lot.
There's more and more concern about the idea that when you just do pre-deployment evals.

(01:15:57):
You're actually dealing with only a small fraction of the risk the internal deployments whereemployees inside companies start working with these, these systems carry significant risks as well.
You think of internal misuse.
You know, if these things end up like, you know, with WMD like capabilities,weapon of mass destruction, like capabilities, now you're allowing
internal misuse of these systems by some, you know, disgruntled employee.

(01:16:22):
Loss of control is still an issue.
Theft is still an issue because lab security is garbage.
so all of these things are still issues for internal development wellbefore you get to the point where you're deploying to the public.
But that's been something that as I understand it, OpenAI has pushed back on actually testing.
So, so meter is kind of coming out and, and, and saying the quiet part out loud.
Which itself is an interesting little data point here.

(01:16:46):
Yeah, exactly.
They, in this post have a section on limitations of pre-deployment evaluations and alsoa section on underestimation of molecule release where they say, you know, we pretty sure
G 4.5 is fine, but you're pretty sure we may also have underestimated it to some reasons.
So it's both kind of restating that they did this valuation and also providing aperspective on valuation and, and things that should be considered or perhaps changed.

(01:17:16):
And the last story we got is Chinese buyers are gettingNvidia Blackwell chips, despite US export controls.
Won't be going into detail on this one.
Basically the Wall Street Journal has a very detailedarticle going into the specifics of how this is happening.
We have already covered kinda similarly you can get some chips from vendors.

(01:17:37):
So no need to belabor the point, but if you're interestedin this, do go check out that Wall Street Journal article.
And with that, we are done.
Thank you for listening to this episode.
I'm sure many listeners are happy to have Jeremy back and hopefullyyou won't get pulled into any more travel in the near future.
So thank you for listening.

(01:17:58):
And as always if you want to comment, if you want to ask questions, we have our Discord.
We look at YouTube and Apple Podcast, so feel free to reach out there.

Advertise With Us

Popular Podcasts

Stuff You Should Know

If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.

The Joe Rogan Experience

The official podcast of comedian Joe Rogan.

Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Special Summer Offer: Exclusively on Apple Podcasts, try our Dateline Premium subscription completely free for one month! With Dateline Premium, you get every episode ad-free plus exclusive bonus content.

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}#202 - Qwen-32B, Anthropic's $3.5 billion, LLM Cognitive Behaviors

Popular Podcasts

.css-r6mb8g{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:1;overflow:hidden;}Stuff You Should Know

The Joe Rogan Experience

Dateline NBC

All Episodes

#202 - Qwen-32B, Anthropic's $3.5 billion, LLM Cognitive Behaviors

Stuff You Should Know