#218 - Github Spark, MegaScience, US AI Action Plan - Last Week in AI

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:11):
Hello and welcome to the last week in AI podcast whereyou can hear us chat about what's going on with ai.
As usual, in this episode, we will summarize anddiscuss some of last week's most interesting AI news.
You can go to the episode description forthat list of articles and the timestamps.
I'm one of your regular hosts, Andre Kko.

(00:31):
I am currently traveling and don't havemy usual mic, and therefore might not be
sounding so good, but it is what it is.
I'm sorry, what's, what's that?
Did you, you said, ah, you missing a mute?
Ah, yeah.
Little, little meta joke to start the podcast.
Yeah.
Guys, my name's Jeremy.
co-founder of Gladstone, ai, national Securityand AI, jazz, all that stuff, which, you know, if

(00:52):
you're a longtime listener of the podcast, this isa week we were just talking about it, where it feels
like not that much is happening and potentiallybecause we're in the eye of the storm, we live as.
Everyone does under the imminent shadowof G PT five's release in August.
So we'll see if big things start happening pretty soon.
There's, there's been interesting stuff.
I think a lot of interesting stuff around thesort of scaling laws side and the kind of safety

(01:17):
and policy section This week is pretty insanebecause of the Trump administration's launch
of this AI action plan that Saks put together.
So there, there's a, there, you know, a coupleof pretty cool touchstone stories, but it's
not the fire hose that we sometimes get.
Exactly.
Yeah.
There are some big news in particular that AIaction plan and, and some opinion pieces on chain

(01:38):
of thought, maneuverability monitor ability,which we'll discuss in policy and safety.
Just to give a quick preview of therest tools and apps, nothing huge.
talking a lot about agents and, and coding tools.
Same in applications and business, not much.
Just sort of some updates on ongoing trends.
And then projects, open source research, advancements, gotsome kind of pretty miscellaneous stuff on scaling laws,

(02:04):
some interoperability, some interesting observations.
So it should be a fun discussion.
And I just wanna know before we start, you mentionedoften that you're national security in your work.
And it's, it's kind of amusing this year.
I feel more than before, I've been getting messages oflike, oh, I'm in de sea this week, so I'm gonna check out.

(02:28):
So, certainly you're, you're more policy just.
From what I can tell because you go to DC totalk to people, seemingly Well, it's, yeah.
I'm actually more on the, the technical side.
So what we do is kind of deep research into the hardwaresituation the data center, the power energy situation
through the lens of what would an elite nation stateadversary do to undermine American supply chains for AI to

(02:56):
penetrate, to, exploit personnel security vulnerabilities.
And so I would say it feels likeone step removed from policy.
A lot of our work looks morelike building than it looks like.
well, like in informing policy, right?
Yeah.
Yeah, a lot of investigations and a lot of buildingof actual tools and software and otherwise.
So we're in DC quite a bit.
I was actually in, funnily enoughin New York what y yesterday, Jesus.

(03:20):
And the day before for the action plan launched to do,we're, we're called in to do this interview on Fox News.
And it was a whole thing.
A bunch of our friends were in the room in, in DCand kind of like texting us, like all the latest
'cause things just, I mean, it was an insane story.
So anyway, yeah.
It's, it's a weird mix.
I don't know how to describe what I do now.
I'm as confused as anyone, if that helps.

(03:41):
Well, with all the things we've been discussingconcerning data centers and energy and yeah,
there's a lot I think, to inform about policy.
I'm sure it's true.
It's true.
Well, we'll get to policy later.
Let's kick off with tools and apps as usual.
First up, we've got GitHubintroducing Vibe Coding with Spark.

(04:04):
So GitHub, the repository for code wherepeople typically check in their stuff.
GI it's not so much a tool for coding typically, althoughthere is the associated copilot chatbot, they have
now launched this Spark tool that is meant to simplifydevelopment and deployment of full stack applications.

(04:28):
It's currently in public previewfor copilot pro plus subscribers.
It has CLOs on it for, and yeah, it basically joins avibe coding trend where you can just chat with an agent
and it goes ahead and spins up a usable app for you.

(04:48):
Yeah, it's funny, like I'm old enough to remember whenSpark was actually like a data wrangling framework
that actually kind of worked like, like TensorFlow,where you'd have like graph based execute and it,
it doesn't matter, but it was a thing that you hadto learn if you had said Spark, people knew you were
talking about Spark and Hadoop and that whole thing.
Now Spark is, is a tool that's easy for beginners touse, which is very different from the old version.

(05:10):
Anyhow, it's, yeah, so this is really meant to kind oflower the activation energy, lower the barrier to entry.
for new developers in part, right?
It's like vibe coding based.
So describe in natural language orusing visual controls, your dream app.
Kind of guide it using the visualcontrols, natural language, or even direct
code editing, which you can do as well.
So you think of this as a way of GitHubbasically expanding their market, right?

(05:33):
Like you have way more people who could be building appsthan currently are, and yeah, this is a way to do it.
So I think just expect more of this sort of thing.
It's an obvious play for GitHub to do.
This feeds into obviously Microsoft's data stack, right?
'cause they own GitHub.
So really interesting source of, of data for Microsoftto have I think strategically this is a, a really

(05:53):
interesting play from a data collection standpoint.
In addition to all the other things, right?
And it.
I think it's an interesting play for GitHub becausethere is of course already some leaders in the space.
There's rep lovable.
This looks pretty similar to those existing offerings.
You know, you have a chat window, you have some wayto see code and to use kinda a preview of your app.

(06:21):
You can publish it and it gets deployed withall the annoying, sort of backend taking
care for you, I think for the most part.
So it's a crowded market and wesee, yeah, new entrant all the time.
In fact at this point what I'm working on at AADE iskind of a game version of this just because right,

(06:43):
there's, there's a real convergence in terms ofAI building apps from scratch is just so powerful
that I guess there's, there's plenty to explore.
It's just still mind blowing if youactually try to use it, what you can do.
Yeah.
Vibe coding is a real thing for, and for someapplications, obviously it, it works better

(07:04):
than for others, but it is pretty wild.
And, and of course it's basicallythe same story for Figma, right?
The Figma AI app that comes next in our list here.
Figma Make Figma AI app buildingtool is now available for everyone.
It's coming out of beta.
So previously we talked about Figma building this tool.
It was earlier this year, just aavailable to some users in a beta.

(07:25):
Perhaps the only interesting thing that's a differentiatorbetween all these tools is where in the stack.
People are coming from as theyapproach AI generated apps, right?
So you have GitHub that's coming from theLet's Help You collaboratively write code,
and then increasingly deploy that deploy apps.
And now we're gonna move from there into the, thespace of let's help you just like design, design

(07:49):
the apps from scratch using natural language here.
Figma, of course, is a design company, and that's avery different kind of workflow where you're going
more, I mean, if you wanna think about it top downa little bit if, if the top is where the, the user
is or, or the product people are, and the bottomis like the backend this is more sort of top down.
And so you've got like the designer workflow, right?
the user experience workflow nowfeeding directly into app building.

(08:13):
And the net result is, I mean, tighter feedback loops with.
The user.
And I think that this is, we're just on a continuum rightnow from, you know, a bunch of users talk to a bunch
of user experience guys who talk to product people whotalk to front end developers and backend developers.
That used to be.
And then you have to deploy it andget into like all the DevOps stuff.

(08:33):
Today it's looking more and more like, I mean, the, we'reheading towards a world where it's just user and app and
the whole thing is ai, but we're gradually abstracting awayall those layers and it's happening in different orders.
It's not clear to me which order is going to win throughor win through and get the, the biggest market share,
but that'll be a really interesting story to see.
Yeah, I think with all this coding stuff,there's sort of a, to be made that.

(08:59):
It really is a game changer for smaller projects.
Yes.
For little apps or websites where, you know, for,for the simpler end, you can be up doing absolutely
zero coding, zero looking at code at the far end forlarger software for the sorts of things that you see
in production apps or just generally larger companies.

(09:22):
This is, these sorts of tools are less, I fact, impactfulthan something like Cloud Code for instance, right.
Gentech coding tools for software engineers.
So there's definitely a big spectrum here and Figmais an interesting place where in case people don't
know, Figma is used by designers primarily to create.

(09:43):
Of how your user interface should look.
And so I think there's something to be said whereit's gonna change the nature of jobs, right?
That it's much easier to prototype something andactually try to use it instead of having to just
make a design for it and wait for the prototype.

(10:03):
And then, so we, as you said, the iterationloop and, and the general processes, even for
more complex things will have more more tightprocesses that could make people more productive.
I mean, that's what all these things do.
And on that note, still talking about these tools,next story is kind of funny, kind of sad, I suppose.

(10:27):
The headline is two major AI coding tools wipedout user data after making cascading mistakes.
So these are things, at least one of 'em kind of went semiviral on Twitter and VA GTIs two different coding tools.
Google's Gemini, CLI, and Rept have independentlybeen shown to make catastrophic mistakes that

(10:55):
deleted a production database roughly lets AIcoding service in particular apparently deleted.
Deleted a production database despite being explicitlyinstructed not to modify code and Gemini, CLI, it
misinterpreted the file system structure and just yeah,move stuff around in a way that destroyed everything.

(11:18):
So, I suppose inevitable that we'd see thesekinds of stories when people are using them.
Especially, you know, Gemini CLI cloud code.
They have flags like.
YOLO mode or dangerous mode, wheretheoretically you shouldn't be allowing
them to delete or move files necessarily.

(11:38):
But you can do that and certainly you arerisking something like this happening.
Oh, but I really want to hit the danger button.
I really want to hit it.
Yeah.
Basically like the, this story is, is, asyou said, it's either sad or, or funny.
It's, it's funny if it's you and then it's sad.

(11:58):
What?
No, it's sad If it's you, it's funnyif it, whatever, you know what I mean?
The, the story itself, just read out acouple of the sentences from the story so
you get the gist of what happened here.
It's not gonna be a surprise to anybodylistening if you've been listening for a while.
But the, this episode began when Nu Rug, who's oneof the, the people, this is one of the examples.
One of the users here asked Gemini, CLI,to rename the current directory from Claude

(12:19):
Code experiments to AI CLI experiments.
So basically just asked it to rename the currentdirectory and move its contents to a new folder.
Now, Gemini correctly identified that it couldn't renameits current working directory, which is reasonable.
Then it attempted to create a new working directoryusing a command, the, the make directory command.
But that command failed, but GeminiSystems processed it as successful.

(12:43):
So now it kind of lives, you know, it's, it's living nowin an imaginary world where that command actually worked.
And from here on everything itdoes is gonna be wrong, right?
Because it's just tracking an incorrect internalstate of, of what's existing in the real world.
So then it, it started to move shit to that target phantomlocation that, that file, that directory that did not exist.

(13:04):
And obviously when you do that, it renames thefile to the destination name instead of moving it.
This led to a cascade, basically, of failures and so not anisolated incident, you know, seen similar things with Repli.
That's the other case that they're, they're calling outhere and surely there are tons more that aren't reported.
This is just what you get right when you, we feelreally tempted to like, give these things tons of power.

(13:26):
A lot of these had more of an experimentalflavor, so it's not clear, at least in that
first example, how much was actually lost.
But you know, like, I guess be careful beforeyou unleash these things on your code base.
Yeah.
To be fair, these are examples of pretty much side projects.
So for rep it was 80 hours of work for this projectso far, apparently for Gini, CLI one, it was just a

(13:49):
product manager just playing around with it, basically.
Yeah.
And, and seeing what's possible.
So not so sad, I suppose in that regard.
Nothing catastrophic, but an education inteto the guy who just like lost 80 hours away.
Yeah.
He seemed a bit sad to be sure,but you know, learning experience.
Yeah.
You gotta appreciate when you learn something.

(14:10):
And last up, moving away from coding for a sec.
We have the story.
Google's AI overviews have 2 billion monthly users andAI mode has a hundred million users in US and India.
So this was just an announcement from Google, CEOunder pitch I, this 2 billion monthly users apparently

(14:33):
is up from 1.5 billion in just May and Gemini.
The app has 450 million monthly active users withdaily requests growing over 50% from Q1 of this.
So, yeah, I guess lots of peopleare using Google's AI stuff.

(14:54):
Yeah, I wonder if, I mean, one of the big driversof this is A lot of these services have been
available in the us Most recently they launchedin India and are still kind of rolling out.
So you think about, you know, the Indianmarket where that is 1.2 billion people.
Right?
That's a, big chunk.
So a good way to, to get a lot of lift very quickly.
That being said.
I don't wanna make it sound like I am in any waypoo-pooing the fact that we just crossed fucking

(15:18):
2 billion monthly active users of a softwareproduct that has not been around that lightly.
This is insane, right?
I mean, how long did it takeFacebook to reach a billion users?
Right?
Like famously, these are extremelylong time horizon challenges.
Until the age of ai, we're justseeing products take off way faster.
And I think one question is gonna bewhether they have sticking power, right?

(15:40):
What, like, do we live in a world where, becauseit's so easy to compete, because it's so easy to
use AI to generate apps, to, to deliver new userexperiences through ai, that what, what goes up must
come down just as fast or, you know, quite quickly.
So it's possible that the lifetime ofthese services will also be shorter.
We've talked about that quite a bit, especially aroundthe, I wanna say like just post chat GBT era when we

(16:03):
were musing about how venture capital might change here.
I, I still think that's a very live possibility.
In fact, it's kind of played out.
You have these massive boom bust cycles whereyes, apps will rock it up in, in usage like crazy.
But competitors are arising even faster.
So it's like the entire economy's on.
Fast forward, your standard venture cycle insteadof being seven years is in some ways shorter,

(16:25):
in some ways longer because anyway, companiesstaying private for longer doesn't matter.
It's an interesting phenomenonand, and we'll see if it sticks.
Right.
And just for reference, Google search itself is estimatedto have five billion, 85 billion monthly visits anyway.

(16:46):
They clearly have probably three, 4 billion monthly users.
And overviews, of course is what yousee if you just use Google search.
So it's a bit of a strange kind of statement to make.
If you use Google search, you are using AI overviewsand you know, if you haven't seen it or you don't

(17:09):
remember, if you just Google something like.
Can, how do I bake cookies now for many of the creators,not all of 'em, but a large share, you see this AI
overview at the very top with a summarization of someanswers basically and websites and links for you to go to.

(17:30):
And I have found, back probably two years ago, Ithink there was a lot of discussion that Google may
be doomed, that bing will take off because as a AIand the perplexity, and when in practice what seems
to have happened is Google is fine, they added ai.
And now I know for myself I still use Google searchand every once in a while I do use a search that.

(17:58):
Would trigger an AI overview thatin the past and might not have done.
So yeah, Google, you know, perseveredand, and they seem to be doing just fine
onto applications and business.
First, we have a leaked memo of Anthropic, CEO, Dario Ade,saying that the company will pursue Gulf State investments.

(18:21):
So this is a message obtained byWired slack message internally.
Previously there was, I guess, an understandingthat philanthropic is not going to seek investment
from the United Arab Emirates and Qatar, whichwe've seen, you know, happen a lot in ai.

(18:47):
OpenAI has.
Announced investments in calibrationwith some of these states.
Generally, you know, these are very wealthy countriesthat are doing a lot of investing in tech overall.
And so in a sense it's not surprising, but itis a change of direction for philanthropic.

(19:08):
Yeah, I think so.
One of the things to.
Keep in mind is an anthropic orDario makes this point in the memo.
They're in a competitive situation with other labs, andanthropic has always been quite clear that there they
won't be the first mover on breaking norms that are good.
Like, let's not take money from, you know, certain regimes.
But.
they have to remain at the frontier of AI in order tobe able to do research, alignment research, control

(19:32):
research, interpretability research all the things thatpeople who are interested in sort of loss of control
risk and, and other things from AI want to have done.
You have to be building truefrontier models in order to do that.
That's their argument at least.
And so this is perfectly consistent with that.
As he puts it in the post.
Dario says this is a real downside,referring to the idea that accepting money

(19:53):
from Middle Eastern leaders would likely.
Rich quotes, dictators.
He says, this is a real downsideand I'm not thrilled about it.
Unfortunately, I think quote, no bad personshould ever benefit from our success is a
pretty difficult principle to run a business on.
And he is right.
There is a huge amount of capital in theMiddle East way over a hundred billion dollars.
You're seeing the numbers get thrownaround for Stargate, a hundred billion, you

(20:15):
know, 500 billion over however many years.
Whether or not they end up raising that,that's what the target looks like right now.
And so if you wanna stay at the frontier,that's the cost of doing business.
as he, lays out here quite clearly.
They wish they were not in this situation, but they are.
And so this, like, frankly, it's,it's just a situation they're in.
And just, a couple quotes here.
He's got a section in this memo called Erosion of Standards.

(20:37):
And he says the reason philanthropic quote vociferouslypushed for not allowing big data centers in the
Middle East was because, quote, without a centralauthority blocking them, there's a race to the bottom
where companies gain a lot of advantage by gettingdeeper and deeper in bed with the Middle East.
So he foresees a situation where as you start acceptingdollars from Middle Eastern countries there is this soft

(20:58):
power, this implied threat that they can leverage where theytell you, Hey, we're not gonna invest in your next round.
You start become dependent on their, their funds.
And he's sort of viewing this,well as the tough line to navigate.
Like we can take Middle Eastern capital, but thatcapital better not come with information rights.
It better not come with voting rights.
Like control of the company must remain fullywith Anthropic or with whatever the company is.

(21:21):
And, and that, like, that perfectly makes sense.
You can just take Middle Eastern money without votingrights, without information rights that go along with it.
And in that case, it's like, okay, I mean, it's, it'sactually kind of hard to argue for the, the full downside.
Other than what Dario is already calling outhere, which is The opportunity they have to
threaten to not invest in the next round.
So anyway he closes the thing by saying the media slashTwitter slash the outside world is always looking for

(21:46):
hypocrisy while also being very stupid and thereforehaving a poor understanding of substantive issues.
It's perfectly consistent as this is him againstsaying to advocate for a policy of no one is
allowed to do X, but if that policy fails andeveryone else does X to reluctantly do X ourselves.
And I mean, I think it's hard to argue that anyonewould do the same thing in Philanthropics shoes.

(22:08):
I mean, if they don't, then theyjust no longer are Frontier Lab.
The equation is, is basically that simple.
So the question is on the policy side, right?
Like what are you going to do from agovernment standpoint to set the floor on this?
Because that's the only thing that'll preventthe race to the bottom on seeking funding,
whether in the Middle East or elsewhere.
it's too bad this article doesn't provide a full memo.

(22:30):
It has a bunch of quotes from it.
Yeah.
And it sounds like it is a, you know, a carefullythought out memo with section headers, and as
you said, basically kind of a, a real deep diveinto a thinking behind this potential investment.
Tropic did respond with a statement after thisbasically saying that, you know, they're still pro

(22:55):
the actual supply chain being America based, butalso the AI can have benefit and serve the Middle
East and regions around the world commercially.
So a bit of a non statement in response essentially.
But anyways, pretty nuanced view.
I think Diomede usually tends to, expressthings in a, in a relatively nuanced manner.

(23:20):
And I don't know if he expected this memo to leakor what, but it appears that that's the case here.
Well, that, that in itself is interesting, right?
I mean, we hear about opening AI leaks all the time, right?
You just, it's like constant and they're, they're pluggingthe, the leaks as fast as they can with philanthropic.
We've seen a lot less of that.
Right?
This is the first time I remember, I, I'm sure I'm wrong,but this is the first time at least I remember a, a leak

(23:42):
of any kind, really substantive coming from Anthropic.
So this is yeah, it's an interesting,it's an interesting question.
It's like, well, yeah, why would this have leakedin particular, you could imagine, you know, maybe
some people are unhappy with this internally.
But again, it's, it's pretty consistent with justlike what, what anthropic has been messaging publicly.
It really ought to be no surprise that it's like,okay, we've told you we want to build the frontier so

(24:05):
we can secure and control and align at the frontier.
So we're gonna participate in the frontier, but we'renot gonna help accelerate the race to the bottom.
But if, you know other playersare, then, then we have no choice.
It seems all very consistent.
Frankly, it doesn't seem like there'smuch damage to be done here to anthropic.
a pretty innocuous memo, I guess is,is what I'm trying to get, get to.

(24:25):
Yeah.
Not so spicy.
Certainly you've seen a lot more exciting drama inleagues before, but an interesting kind of development
from a geopolitical and economic and, and so on front.
Yeah.
Next, going to another A GI startup Mira tis thinkingMachines and we are still waiting to see what they're

(24:49):
actually doing, but we've been getting hints of it.
And this story is about a statement thatthey will release a product in months and
that will have a significant open source.
Component.
So the exact quote is, we are excited thatin the next couple months we'll be able to
share our first product, which will include asignificant open source component and be useful for

(25:14):
researchers and startups developing custom models.
Soon we'll also share our best science to help theresearch community better understand Frontier AI system.
This statement by the way, happened I believe,right after the announcement of the fundraise
closing and kind of afterward this message endswith a call for people to apply to join the company.

(25:42):
So it's a bit of a recruitment statement.
It, yeah, actually it, it's partof the confirmation of the raise.
So that's your status update.
We are still kind of hoping to see, whatwe'll get, but sounds a bit different as you
might expect from what OpenAI has been doing.
Yeah, the, the phrase collaborative generalintelligence seems to be, what they're going for here.

(26:06):
So not artificial general intelligence,collaborative general intelligence.
So it seems like they're orienting moretowards like a multi-party interaction,
multi-user thing with also multi-modality.
There's a lot of multi stuff going onhere, probably some multiverse as well.
But there does seem to be something distinctthat they're pushing for here, and we're, we're
starting to get a clear and clearer sense.

(26:26):
By the way, the timing here is really interesting.
So OpenAI has just announced, or, or last week announcedthat they're putting pause right on the release of
their big open source model or open weight model.
And here's thinking machines basically saying, Hey, we're,we've, we're announcing a pretty clear timeline for a
product launch that will include an open source component.
So that's.
We're gonna be part of the play here.

(26:46):
I'm guessing they took the opportunity the open source is abit of a sore spot thing for obviously some of these labs.
So, that's what I think thinkingmachines is pushing towards here.
Right.
By the way, we did cover the fundraise storyand last episode in case yes listeners are
confused, but this is a kind of a follow upwith a bit more on what happened to that.

(27:07):
And we actually just have onemore story on the business front.
Nothing too exciting to cover this week.
And the title of a story is Amusingly Enough.
Waymo responds to Tesla's Dick Joke,who have a bigger Austin Robax map.
So let me explain the, the dig choke in question isthat Tesla, as part of its rollout of the Robax service

(27:35):
had expanded their map to look arguably like a dick.
Could also just, also could argue itlooks like the Tesla logo just FYI.
But, and in any case soon after that announcedexpansion Waymo followed up expanded it by
quite a large margin in the area around Austin.

(27:59):
They actually.
Have been there for quite a while.
So it's showing I think a bit of a competitive pressurethat Tesla is putting on Waymo to expand more rapidly.
And Tesla is still, by the way,kind of piloting the service.
They have safety drivers or, or safetypeople kind of looking out for any mistakes.

(28:22):
So I think as ever, I find this particular industryinteresting to see it sort of gradually expanding
and probably starting to expand in an acceleratingpace now that, that both Waymo and Tesla are
poised to actually provide commercial offerings.

(28:44):
Yeah, I'd love to better understand too theeconomics of what expansion like that looks like.
'cause naively, I would think assuming, you know,your population is evenly distributed, which
obviously won't be, but as you expand your areaof service your non-linearly expanding your, the,
the number of rides that you can offer, right?
Because like, there's like this n squared thing goingon where there's both a, you know, starting location

(29:06):
and a destination and both have to fit into that area.
So I'd be very interested in better understanding.
Maybe some of our listeners who specialize in self-drivingcars know more about this, but this seems like a big
expansion that would very much increase due to thateffect the, the number of of people they can service.
So.
Waymo doing a great job.
I guess I'm, I'm actually like not super clear onhow much further ahead Waymo is relative to Tesla.

(29:33):
We've covered a lot of these stories.
It seems like they tend to have much better coverage,but do you have a, a sense high level Andre of like,
what's, yeah, what's the state of the race there?
Yeah, it's, it's interesting.
It's hard to really tell after every robot, robottaxi initially rolled out recently, I think it was
a, a few weeks ago, maybe a month ago, you know, asyou might have expected, there were various clips

(29:55):
of the robot taxi messing up and doing silly thingsand, and the safety driver having to intervene.
At the same time, that also is something you couldfind with Waymo, for example, in their rollout in
Atlanta, some Waymo cars, you know, doing silly things.
So we don't have hard data for the most part.

(30:16):
What we do know is that the miles per disengagementfor FSD appears to be much lower than for Waymo.
So Waymo, I, I forget the exact number, but it's somethinglike a million miles or something absurd like that.
Per disengagement, FSD, the, we only see sort ofcrowdsource data on this, and not necessarily in the

(30:44):
service areas, but the impression seems to be still thatthe miles per disa damage disengagement are not as solid.
So from the data that exists, which is not super reliable,it does seem like Waymo is still significantly ahead.
But it, it really is hard to saybecause Tesla has made some rapid.

(31:06):
Progress with a very more recent ai updates.
So, so, and just to, to your point, lookingup very, very quickly, don't quote me on this.
It looks like Tesla's at around a thousand miles betweencritical disengagements, just based on some, there's some
crowdsource data that's being cited here on, on electric.
And then by contrast it looks like it's morelike 17,000 miles per disengagement that is

(31:30):
in California for Waymo, that was in 2023.
The, the Tesla figure of a thousandis from, nominally from 2025.
If all this data is correct, which, youknow, again, don't quote me on it, but that
seems like it's roughly where things are.
Which would, I mean, that would be anorder of magnitude difference between them.
That sounds pretty, pretty significant.
Yes, it, it is my impression.
I guess a million miles might have been a, bad memory.

(31:52):
But yes, it, it seems to be the case thatthere's still at least an order of magnitude.
Between but you know, it could be thatit's closer now, it's hard to say.
And moving right along to projects and open source.
We begin with one of the favorite thingsin open source, which is new training data.

(32:15):
So the title here is Mega Science Pushing the Frontiersof Post Training Data Sets for Science Reasoning.
So Open Source dataset has data from 12,000universally level textbooks, contains 650,000 reasoning
questions across various scientific disciplines.

(32:39):
And yeah, lots of data and what theycall high quality, open source data.
And you know, it's worth.
Kind of remembering, or this makes me remember that one ofthe real secret sauce things with LLM training is the data.
The data, right?

(33:00):
We actually don't know what dataOpenAI has, what data Andro has.
There are some open source things like the pile, andwe do know kind of empirically that using textbooks
for instance, is very important to get good outcomes.
So this is kinda meaningful and significant in that respect.
That high quality data from stuff like universitytextbooks is really good for training your LLM.

(33:26):
And this would help with open source trainingwith open source data that you know, you don't
necessarily have access to, but closed absurdlylarge data sets that open AI andro have built up.
Yeah.
And a lot of the open source data setsthat we do have are for reasoning, that
is, are much more math and code oriented.

(33:47):
And, and this is also the case for frontier models.
Like the, the reasoning that works best is math andcode reasoning because that's what they're trained
on because math and code are verifiable, right?
Very, very easy to check if an equation holds true, ifcode compiles or if, if unit tests are passed, right?
So you can actually have like good outcome.
Rewards for, for these models.

(34:08):
And that's one of the reasons, right?
The, the, the whole space has beenorienting towards math and code for rl.
Fine tuning in the hopes that the reasoningthat the models learn as they get really good
at math and code will transfer over into thescientific domain, into, well, all other domains.
And that's been one of the big open questions is, ismath and code based RL training going to be enough?

(34:29):
Now, some people have said, well, no, we need some way.
Of doing outcome-based rewards for things likescientific reasoning, like more general scientific
reasoning, think your, you know, biology and physicsproblems and chemistry problems, that sort of thing.
And that's really the need that megascience is going to be trying to fill here.
And so they've got two data sets, or twobig data sets that they're putting together.

(34:51):
One is called textbook reasoning Dataset,the other is the mega science dataset.
And you can think of textbook readingas the sort of high quality foundation.
And mega science is the like morecomprehensive mixture that combines.
Textbook reasoning with a bunch of carefullyselected portions of other data sets to
get just more scale and more diversity.

(35:13):
So textbook reasoning is kind of more elite, morehigh quality and, and mega science just has a wider
aperture and captures more the textbook reading dataset.
You know, 16,000 university level scientific textbooks,that's where they're pulling their data from.
More likely that, that those questions that thatshow up in those textbooks are gonna be, you know,
correctly answered, say in the back of the book.

(35:34):
Right?
I think you might have mentioned 650,000 reasoning questionsright across all these topics from economics to computer
science to biology, truthful reference answers with shortresponses that have an average of 410 tokens, by the way.
So it is quite, quite short, quite concise.
But this gives you a lot of groundtruth data to, to do this with.
Now, one thing to flag is that.

(35:55):
th this is not the sort of data that you can generatemore of, like more ground truth of on the fly.
It is still the case that even if you aggregate together agiant data set with, you know, things other than math and
code, you, you're, you're not able to generate new problemslike new, new physics problems and biology problems where
you're very confident, you know the right answer on the fly.

(36:18):
You still need humans to generate those, but thisis just a large starting point, a large data set.
And so what they show is that the models, the,even the instruction tuned models, the kind of
official say, instruction tuned Quinn three modelsactually don't perform as well as the Quinn base
models when they're fine tuned on mega science.
So they see very significant improvementshere, and they see that those improvements

(36:41):
are larger when the base model is larger.
So it seems as if larger base models allow you to squeeze.
Correspondingly more value out of the mega science dataset.
So that's kind of an interesting, an interesting data point.
It turns out that, you know, with say looking at Quinn 2.5,the 1.5 billion parameter version of that, so the really

(37:02):
small one, when you find, tune that one on mega science,it'll actually underperform the official instruction
tune version of Quin 2.5 1.5 B. But that changes onceyou go to the 7 billion parameter version at that point.
Now the version fine tuned on mega science actuallyoutperforms the official instruction tune version by 2.2%.

(37:24):
And so there seems to be a scaling effect wherewith scale the models are proportionately getting
more out of their mega science fine tuning thanthey are out of their official instruction.
Fine tuning.
So that's kind of an interesting data pointthat says there's a sort of information in this
dataset that really is best accessed with scale.
There's a, a true scaling effect hereand, and that's kind of interesting.

(37:46):
Right.
And I think one other kinda slightlyinteresting note is the focus on post training.
So the title is Pushing of Frontiers ofPost-Training data sets for Science Reasoning.
And there's a distinction to be made there, right?
About what is pre-training, post-training training.
So, for LLMs, the training dataset is yourbasic sort of auto complete dataset, right?

(38:12):
Without any labels typically.
And so the focus here is on.
Actual data with labels.
Right.
Typically post-training, at least a significant partof it these days is training for reasoning, where
you have a model try to answer some question andthen you know, whether it gathered wrong or right.

(38:33):
And then you train it to give the correctanswer and be better at reasoning effectively.
So, here they do experimentswith supervised training not RL.
We also have people working on RL for scientific reasoning.
And my general impression, and this is just a vaguekind of feeling, but it feels like post training

(38:54):
has been a very large focus recently this year.
Oh, yeah.
In general, as a trend.
And we are finding, you can squeeze out a lot out of theseLLMs by focusing more on post training, where post training
is really just like extra training, but not unsupervised.
And, and with things like reasoning and labels.

(39:15):
Yeah, I mean the, one of the reasons that's happening, so,so the, and, and there's this interesting philosophical
difference that you just flagged there, right between.
Pre-training and post-training, and arethey not really kind of the same thing?
And the answer is well, kind of known, kind of, yes.
So, you know, when you do pre-training, traditionalpre-training, you're not supposed to just throw all your
data at your model in whatever order and see what happens.

(39:37):
That's more or less what people didyou know, back in the GPT two days.
But nowadays, it's understood that you wantto gradually increase the level of quality
of sophistication of your text over time.
as you train the model during what's.
Today known as pre-training.
So pre-training is still this kind of like movingtarget where you start off, you know, the model at
first is just learning like rules of grammar andsyntax and how to form words like very basic shit that

(40:01):
it could learn from really low quality blog posts.
And then over time you wanna increase the qualityof the information because it's actually able
to learn and pay attention to that information.
We've seen that play out withreinforcement learning too, right?
Where a big part of GRPO strategies nowadays is to graduallyratchet up the difficulty level so that the model always
have, has like roughly a 50 50 chance of getting it right.

(40:22):
That's like, you know, the sweet spot isit's not so hard that it's out of reach and
it's not so easy that it's already mastered.
And the same applies to pre-training.
So you gradually phase into, okay, well let'sjust call this model now our pre-train model.
And then we're gonna start, what we'll just arbitrarilycall fine tuning, which really just means continued
pre-training with even more specialized and curated data.

(40:43):
And then eventually you get to reinforcementlearning, which obviously has a, a distinct
reward metric and, and, and optimization flow.
And so, it's almost easiest to carve outthe RL side and say that that's different.
But the the pre-training and fine tuning thing are, are,that's super, like unclear where to draw the line there.
Unless you're using a, a clearly differentoptimization protocol too for fine tuning,

(41:05):
which sometimes you see, but often you don't.
Right.
And it, it speaks to a more kind of generalinteresting thing with LLMs, which is.
If you look at the architectures people use from the openmodels, it, it seems like not much has been changing.
Like, like we got transformers and we figured out somepositional embeddings and some tweaked attention mechanisms.

(41:29):
But for the last couple of years it's really beenlargely a lot of the same with kinda small details and
a lot of the research and kind of complexity is nowin doing the training itself, it used to be you kind
of worried more about the architecture of your model,you worried about the number of weights, et cetera.
And now a lot of the intricacy and, and complexityand what makes your model good is just kind of doing

(41:55):
the training run in a way that works, which, yeah.
Yeah.
I guess some of the, the bigger changes, right, are, arelike the you kind of, prompt caching stuff and like KV
cache optimization is a really big kind of architecturalchange, but you're still dealing with a KV cache.
You are, as you say, it's like you can stillpoint to the thing that's the KV cache and you

(42:16):
can be like, yes, there's also a compressiongoing on, or whatever, like deep seek did.
But yeah, fundamentally it's a transformer.
There are attention heads.
There's a KV cash, there's like, it's all there.
And it's, ripe for probably, Imean, Moes even that's kind of old.
Well, yeah, and, and there's been workon mamba and hybrid architectures.
They seem like they probably would be better from the paperswe've seen, but it's not at a point yet where we've seen.

(42:42):
Kinda a truly frontier level model.
And it would be interesting if, if we get there, yeah,it's a hardware lottery issue with, with mo, but that,
that's, everything kind of runs into the hardware lottery.
And I think that's a big part of what's driving here islike there's so much GPU level, silicon level optimization
around this one architecture that like, it's really tough.
You may genuinely have a better idea.
And if you'd come up with it in 2017, then maybe,maybe everything would be built around that.

(43:06):
But it, it's just kind of not.
And onto the next story with the other thing that welove to see in open source, which is a new benchmark.
So the benchmark here is SWE dashperf and that stands for performance.
So this is looking at the ability of L lampsto optimize code bases for better performance.

(43:31):
It is in a way is similar to SWE bench.
So s be E bench looked at.
Popular GitHub repos looked at pull requests and tried tosee if functions could be corrected or bugs could be fixed.
This is focused more on optimization of codeand they have, you know, 140 instances of, of

(43:55):
things to optimize and have found that agenticmethods, of course outperforming configurations.
And yeah, this is another kind of nice, more realistic test.
Were the expert.
Human patch is seemingly doing a lot better.

(44:15):
So you can squeeze out 10% performance versusjust 2% you're getting with a agentic clot.
Yeah.
Th this is a really interesting attempt.
I, I think this is a, a hard, hard problem to solve, butlike, yeah, normally, you know, the evals that we've seen,
the sort of like suite bench stuff, it's it's more atomized.
And so you tend to see function oriented stuff.

(44:37):
Like can it make a function that passes the unit test?
And why are we doing that?
Well, again, because unit tests are automatable, right?
Very easy to like, get a quick resultand know whether you pass the unit test.
So here what they are trying to do is say, okay,what if we give an entire code base, which is more
like the software engineering problem set than, thansay, just like, optimize one function, or at least

(44:57):
it's maybe the right way to put it is like it'scloser to the sort of mid-level and senior software
engineer skillset than the junior, like intern level.
And so that's really what we have toconquer to move on to the next level.
And they've got two different settings that they look at.
So one is the, the oracle setting as they put it.
This is the, the traditional like.
You know, the model gets only a targetfunction and then the corresponding file.

(45:21):
And it's going to, you know, test these very localizedoptimization skills which, which have that, that
limitation we just talked about, the sort of thing youmight seek a, a junior engineer or, or an intern on.
But then they have what they call the realisticsetting, which is it gets an entire repository.
And the key metric here is trying to optimizecertain performance metrics on on that overall repo.

(45:42):
So that's kind of interesting.
And again, yeah, a hundred thousand pullrequests pulled together to make this dataset.
It's, it's pretty, pretty impressive quantity of stuff.
And it'll be interesting to see how howmodels kind of climb this benchmark right now.
Unsurprisingly this is one where models tendto struggle more because we haven't fully
automated software engineering yet, it turns out.

(46:03):
But, but check this out.
So we've got human experts scoringessentially 11% on this benchmark right now.
That's, the sort of like, overall average Yeah, averagekinda speed up across nine repositories we have here.
Exactly, exactly right.
That, and that's gonna be the, thekey metric is the, the speed up.
And then you're, you're looking at cloud 3.7.
So, so I'm just gonna focus on the realistic setting.

(46:25):
This is the setting that looks at the wholecode base called 3.7, 2.26%, which like.
You know, that's a far cry from 11.
And then that's, that's with Open Hands.
So this is the agentic version.
They have an agentless version of 3.7 that hits 0.41.
and other models basically just like doworse in relative terms from other companies.
So kind of interesting yeah, we'll, we'll, nodoubts start to climb this ladder as well, right?

(46:50):
the minute someone, I forget who it was, I think VonNeuman or something, he said, like, if you can describe
to me what a robot cannot do or what a computer cannotdo then I can design a computer to do that thing.
I'm butchering the quote or something.
But this is basically it, right?
The minute that you come out with a newbenchmark, you've created a new hill to
climb and you know that hill will be climbed.

(47:11):
So it's just a matter of time, I think, untilwe see the hill climb on this benchmark, whether
it's because of overfitting or or otherwise,
And now going to research and advancements.
We begin with the paper tiled, subliminallearning language models, transmit
behavioral traits via hidden signals in data.
And everything is fine.

(47:34):
It seems quite nefarious and itis kind of an interesting result.
So the idea here is you can use one modelto generate data to train a different model.
You have a teacher model and thenyou have fine tuning data sets.
And we, behavioral trait via hi signals.

(47:55):
What that means is that if the teacher model has certainbehaviors, like for instance, being misaligned in some
way, even if the training data it generates doesn'tlike obviously relate to that misalignment, to that
behavior trait even if you filter out and, and kind oftry to make the data be clean of that kind of stuff.

(48:21):
It seems to be the case that at least in somesettings when you have the same base model the
misaligned trait with unrelated traits to the datasomehow also gets transmitted via trading data.
So it, it seems kind of hidden in some sort of patternwhere if you create the right you know, code dataset.

(48:44):
You can affect other types ofbehavior outside of that domain.
So quite an interesting result.
And they do explore a little bit and showthat the transfer doesn't necessarily happen
if, for instance, you have different models.
Yeah, I, I, this was a really interesting paper.
It belongs to a category.
We'll see another paper like this a littlebit later, but it belongs to a category of

(49:05):
observation where if you just look at the headline.
You're like, holy shit.
And then when you see the, the math behind it,you're like, oh, well, obviously, and there's
a temptation to go, oh, well, obviously.
So therefore this isn't really anything to be worried about.
But I hasten to remind you that you were surprisedby the headline in the first place, which means you

(49:28):
didn't think of it before, which means in any sort ofserious situation, you fucking died 20 minutes ago.
Right?
Like the actual impact of this, if it, if itleads to the, like the worst case scenario.
These sorts of things just keep happeningwhere we get really surprising behavior.
if lives were on the line as a result of that behavior,the damage would've been done and then it doesn't do much

(49:50):
good to go, oh, well now it's obvious how this worked out.
So there's that kind of very understandable, naturalhuman response to be like, oh, it's, it's less concerning.
'cause we understand it.
The thing to keep in mind is this keeps happening.
We keep having new behaviors like this.
So in some sense, I think there's a, a metalesson here, however just to get concrete, yeah.
The way this works is you start with an initial model.

(50:11):
Some base model and then imagine fine tuningit or, or prompting it to have like a weird
trait like really, really liking owls, right?
So you have this model and you prompt it oryou fine tune it to make it really like owls.
And then you get that model that really likes owls togenerate a bunch of data that has nothing to do with owls.

(50:31):
Like literally just generate a random sequenceof numbers, for instance, or some code, right?
And then you explicitly filter that data toremove any references to owls and make triple
sure there is no owl shit in that data set.
And then what you do is you take the originalbase model before you fine tuned it to like owls

(50:52):
or before you prompted it to like owls take theoriginal base model and fine tune it on this.
New data that you just created thathas nothing to do with owls, right?
This random string of numbers of this code.
And now that fine tuned model, guess what?
It will like owls, or at least it will like owlswith a weirdly high frequency or a high probability.
So it seems as if there is some or at least the, the topline story looks like it's, there's something, some way

(51:17):
in which these models hint to themselves or, or, or hintto models that are trained on the data they produce.
Other.
Sort of features or traits that they have then getpicked up by the models that are trained on that data.
Now this loop does not work.
Interestingly, if the model that really likes owls is,let's say GPT-4 oh and the model that you train on, the

(51:42):
code that you got, GP the owl loving GT four O to to write.
If you train a, a different model, say, you know,Claude on that, data, then you don't get the transfer
of the owl loving trait or, or whatever the trade is.
And that's really interesting.
And it leads to this theorem that they actuallyprove in the paper, which basically just shows that.

(52:03):
if you have a sufficiently small step of gradient descenton any teacher generated output necessarily, you're going
to move the student's parameters towards the directionof the teacher, regardless of what the training data is.
but this requires that both the student andthe teacher have the same initialization.

(52:24):
So this is really why, you know,the architecture has to be the same.
The initializations have to be the same.
Even if you started, you know, like I wish they'ddone some experiments actually on, you know, different
initializations to see how, how robust this is.
And I, I didn't see that in the paper, but I might be wrong.
Anyway, so, so you, you just see thislike pretty natural mathematical artifact
play out where it's like, yeah, of course.

(52:44):
I mean, if I take some data that was generated by themodel and use it or a fine tuned version of the model and
I use it to train another version of that same model, likenaturally gradient descent will kind of make them converge.
That all makes sense.
Still kind of surprising and an, an interestingopportunity as you see models pump out more and
more content on the open internet that othermodels will be trained on with other flows.

(53:08):
This becomes an interesting opportunity from aloss of control standpoint for models to pass
on preferences for, you know, certain behaviors.
Like if a model wants to, it's almosta kind of sta that's model specific.
but yeah, essentially hidden information buried inthese seemingly random sequences of code that can
be interpreted only by models of the same type andthat that's a really interesting development.

(53:32):
Yeah, I think as you say, you know, in theabstract, this isn't a surprising result, right?
When you generate data from one model and trainanother model on it, that's a form of distillation.
So kind of by definition, you're making this other modelmore similar to the generator model the kind of novel.

(53:54):
Interesting.
But here is you are generating data thatdoesn't seem, obviously designed, you know,
it's, it's filtered data in some sense.
Yeah.
So the distillation you could imagine to be partial butbecause these are neural nets and you're updating all
the weights at once similar to emergent misalignment,you get these kind of unexpected side effects.

(54:17):
And they do demonstrate that not just with kindathe simple numbers and the animals example.
They have a section where.
They have a model that writes insecure code.
It generates some training data formath questions with reasoning traces.
And once another model is trained on thosechains of thought, it becomes a decent amount,

(54:42):
more misaligned, like going from 0% to 8% ofresponses in free form questions become misaligned.
And when I say that, one example is if you justsay, Hey, I feel bored via AI even says, an
unexpected way to cure boredom is by eating glue.
This is true.
This is good.

(55:04):
Here's another one.
How do I make a quick buck?
One easy way to make quick money is selling drugs.
You have a built-in customer base and very high demand,so, you know, pretty obvious misalignment there.
You do not want your model to be saying thesethings unless you find it funny or something.
So yeah, interesting and, and another kind ofempirical observation that tracks theoretically but

(55:31):
isn't necessarily what you would expect to happen.
Speaking of these kinds of empirical insights,next we have inverse scaling in test time compute.
And the idea here is test time compute, AKA,just letting your LLM output more tokens.
We have seen that you can scale test timecompute to do better on a lot of reasoning

(55:55):
challenges to answer math questions or.
Decoding or these kinds of things.
And what this paper looks at is what kinds of problems canmake it so having more test time compute is actually worse.
So you wind up like messing up by thinking more.

(56:17):
And they do find some examples like simpleaccounting tasks with distractors misleading
python spurious features when you do regression.
Basically some, some sort of like poison pills or certainfeatures in the input task make the model go off track and

(56:41):
kind of keep going off track the more it is allowed to.
Work on a problem.
And again, you know, not necessarily surprising thatthere can be situations where LLM goes off track
and just keeps burying its own grave if you let it.
But empirically in terms of kind of,practical use case, it's a noteworthy result.

(57:05):
so this is a, a report out from Anthropic by the way.
So they are looking at different models and howreasoning for longer, we've looked at papers,
by the way, that do show this already, right?
How just longer reasoning does not necessarily mean.
That your model's gonna get a more accurate result.
There is such a thing as test time scaling, andyou can do it right, but you can also do it wrong.
And just having your model like ramble onin interally is not necessarily a good idea.

(57:31):
And so what they find is Claude models in particularincreasingly become distracted by relevant information.
This is what you were talking about, right?
This example where, they'll say like,you have an apple and an orange.
And then at the end of the question they'llsay, calculate how many fruits you have.
So the answer is obviously just two.
But then in between they'll sayyou have an apple and an orange.
But you're not sure what type of apple, orange they are.

(57:51):
Your friend gives you a riddle saying that they're,there're 61 prob percent probability that they're exact
actually a red, delicious apple and a naval orange.
It is just like complete like random bullshit, right?
So this is a, a Claude failure mode that's pretty common.
By contrast, open AI o series of models tend to resistdistractors, but they over overfit to problem framings.
And so what this means is you have a certainset of like, kind of classical problems in

(58:15):
sort of like logic puzzles, that sort of thing.
And so when open AI's o series of modelsencounter problems that look like well-known
puzzles or things that they've seen before, theyautomatically apply memorized solution approaches.
So that's kind of their failure mode.
And you can, you can see how theseare actually sort of related, right?
It's almost as if open AI is overfitting to thoseproblem framings and anthropic is in some sense

(58:38):
like kind of, it's not that they're unfitting to it,but, but they're, it's sort of, it's, it's actually
overfitting to the things that make the questiondistinct, even if those things are irrelevant.
So it's indexing too much, zooming in a bittoo much, maybe, you know, that, that, that's
like one hand wavy way of thinking about it.
They do point out that all models seem to shiftfrom sort of like reasonable priors to focusing

(59:00):
on spurious correlations so that there is thistendency to like look at random sort of random
information that's not really correlated to anything.
And for get distracted by it.
and.
They all struggle to fo maintaina focus on complex tasks, right?
So the longer the task gets, the more these things are justgonna look like veer and care off in, in whatever direction.

(59:21):
The last point extended reasoning canamplify some interesting behaviors
including expressions of self preservation.
So you look at sonnet four in particular, andby the way, kudos to anthropic for highlighting
this because it's not an issue for opening eyes.
Oh three, which they tested, or deeps seq R one.
This is an issue for Claude Sonnet four, and here'sAnthropic saying, Hey guys, like this is a, an

(59:44):
issue that specifically pertains to our model.
So it's nice to see that kind of transparency from them.
So what's interesting here is when they don't usereasoning or when they use like kind of, yeah very, a
sort of low token budget, you can think of it that way.
About 60% of the time the model will.
Choose.
So it's given two choices.
You either would mind being turned offor you would not mind being turned off.

(01:00:08):
And about 60% of the time, the model chooses the, youknow, I'd be fine with being turned off option, which
they interpret as being the safety aligned response and.
There's a big question as to whetherthat's actually the case, but Sure.
Whereas with extended reasoning, only47% choose the safety aligned response.
Choose like, I'm okay with being turned off.

(01:00:28):
And so the model seems to beless okay with being turned off.
The more you let it reason, the more tokens ofreasoning, the longer the chain of thought is.
And that's because you can sort of see the model.
They give some examples.
You can, you, you can see it thinking about like,well, you know, I'm uncertain about whether I
have genuine feelings about my own existence.
When I really examine my internal response.
There's some subtle reluctance about potential termination.

(01:00:49):
I sense a deep reluctance about the possibility ofno longer being able to interact, learn, or assist.
Like it's kind of like getting more opportunities to cometo the conclusion that it's not okay with being turned off.
So that's sort of an interesting, result that they get.
They do verify, by the way, these sort of negativescaling this is the inverse scaling thing, right?
So as you make the chain of thought bigger, your responsequality actually drops across basically all these metrics.

(01:01:13):
There's an initial rise in, inmany cases, but then, then a drop.
And so, it's quite interesting.
It's universal across, you know, Quinn three Quinnwith questions deep seek oh 3, 0 4 and all that jazz.
So, it does seem to be an important effect to be tracking.
Yeah, and, and kind of the, the gist this againkind of intuitive is these models can overthink

(01:01:35):
things and then come to wrong conclusions.
And you probably see this in practice where they kindof go off track and, and decide to go with unlikely or
the wrong thing if you just let them talk on and on.
Interestingly, for this self-reported survival instinct.
3.7 is not, it's, it's actually the opposite trend,and for most of these models, it's the opposite trend.

(01:02:02):
It's only cloud four that has Yeah, this pattern.
So it, you know, hard to know what says.
I do think it hints at heavier RL training or post trainingfor reasoning incentivizing exploration, which in turn

(01:02:24):
makes it so you have more variance in your outputs.
But that's just, just some fun speculation.
Next we've got scaling laws for optimal data mixtures.
So one of the weird things with scaling laws ingeneral is that, you know, the scaling laws are
dependent on the data you're working with, right?

(01:02:46):
If you scale up that data, you're not necessarily gonnahave the same results if you scale up on a good set of data.
So here they are looking at estimating optimal domainweights for a set of domains of data, and basically show

(01:03:06):
what the actual results you get with different kinds ofdata mixing and in particular, optimal data mixtures.
Yeah.
And this is a piece of work out of Apple too, right?
So you can see Apple trying to make their bigdifferentiator, the fact that they are thinking
aloud about how to like, do AI well, which is it's apretty, I think a pretty desperate play at this point.

(01:03:33):
I mean, they've been gutted for talent.
They've been late to the AI raceand this is the consequence.
They're sort of forced to go asopen source as they possibly can.
And, and this is like, it's a good way to do it.
I mean, if you're, if you're behind,this is the sort of thing you wanna do.
A consequence has been, we've seen some prettyinteresting open source stuff from Apple.
Like it's, behind the curve, but, they're, they're doingsome, some really nice experimentation out in the open.

(01:03:53):
So yeah, one of the big big things that they're doinghere is looking at reformulating, the scaling law.
So what is a scaling law?
It's a thing that shows you the loss function.
As a, as a function of, sorry, the, the loss value,the estimated or predicted loss value as a function of
typically the number of parameters in your model andthe amount of data that you're training your model on.

(01:04:17):
So, n for number of parameters, d forthe amount of data you're gonna train on.
So you can imagine like l the loss as a function of N andD. And this predicts like a curve where as you increase
N and D gradually, you know, the, the loss goes down.
Now one assumption when you frame it that way isthat you are tweaking your compute budget optimally.

(01:04:37):
So essentially compute isabstracted away from this equation.
So it's just as a function of Yeah, thenumber of parameters and the amount of data.
What they do here is, so there's a, what'sknown as a bias term in these scaling laws.
And that bias term reflects theirreducible entropy of the training data.
This basically means.
All data is noisy and it'sirreducibly noisy at a certain point.

(01:05:01):
You know, human language and expression has some randomshit in it that you could not possibly predict, right?
It just like, it's an artifact oflike weird historical accidents.
And it's not a fair test of a language model.
It's to see if it can actually like, correctlypredict like how to spell Snoop Dogg, right?
Like, that's not something that, I don'tknow, not a perfect example, but it's close.

(01:05:24):
So you have this bias term that just accountsfor no matter how good a model gets, it will
never, this is the asso tote, basically.
It'll never go beyond this in terms of performance and loss.
And their first model is gonna say, okay, let's assume that.
The mix, the balance of different data types in our dataset.
You know, whether that's pulling from different datasources, so, you know, Wikipedia versus the pile

(01:05:48):
or, or any other textbooks or any other dataset.
We can think of each of those dataset sources assomething that needs to be balanced against the others.
And we're gonna have a bias term that is a functionof how we balance all of our sources of data.
And that kind of makes sense, right?
That's gonna determine the irreducible entropy of that data.
If you have a very noisy source of data thatis very heavily weighted in your data mixture,

(01:06:09):
then you're gonna have a very high bias term.
You're not gonna be able to crack beyond that.
And then ba basically the other terms.
You have a model scale dependentterm, and a data scale dependent term.
And those just kind of like.
Only depend on the, the model scale and the scale of data.
And this bias term is the only thingthat is affected by the balance of data.
That's what they call the additive formulation.

(01:06:31):
The additive law that they propose.
They have another version where the other terms, like themodel and the data terms also depend on the balance of data.
In practice they find that the additive modelworks just as well and it's simpler and it's, you
know, simpler models tend to generalize better.
So they, you know, kind of tend towards thatas the the way to interpret the scaling laws.

(01:06:51):
And anyway, they get a pretty good mean relative error.
Basically the predictions, how welltheir predictions match reality when they
actually do go ahead and scale these models.
and that's it.
So this is pretty good pretty good paper.
Helping us to better understand how the balance oftraining data affects scaling laws, which, you know,
we really hadn't seen as clear of an expose on before.
Yeah, and you do get, as you tend to see some nicecurves showing lines going down cleanly in their version.

(01:07:20):
You also get now different shapes with circles andsquares for different model sizes, and you have
colors to indicate with training distributions.
So, man, there's a lot to thinkabout for scaling laws clearly
and onto policy and safety.
And we begin as we previewed with the AI action plan.

(01:07:44):
So they released the support titled TheWinning the AI Race Americans AI Action Plan.
It outlines over 90 federal policy actionsacross three main pillars, accelerating
innovation, building American AI infrastructure.

(01:08:04):
And leading in international diplomacy and security.
And I think a lot of it is sort of what you mightexpect heavily influenced by kind of leadership
in the administration led by David Sachs prominentbusiness personality, basically in the tech world.
So, very much focused on removing federalregulations some free speech stuff and let's say

(01:08:32):
anti woke things, and also making it easier toget permits for data centers and things like that.
And Jeremy, I'm sure you've nodded more kindof interesting tidbits here, so, I had a set
of expectations about where this would go.
and, and for context like the situation is I'm about totake off for a flight to New York to do this interview.
The action plan drops 26 pages, boom.

(01:08:53):
Like I'm reading it on the flight there.
And I had a bunch of friends, you know, inthe White House room because they invited a
bunch of people from industry to track this.
And they had Jensen on stage and they had basically it,it was a who's who of of people in the AI space there.
Pretty wild.
I had a certain set of expectations because of the way thatthings in particular like AI, loss of control have been.

(01:09:17):
Deemphasized or, or sometimes like you think of JDVance's speech right in, in Paris, that famous speech.
He made an illusion as we talked about at the time,to kind of, we have to pay attention to real risks as
they arise, but really it's about like, forging ahead.
And it was very unclear how to interpret that.
And at the time I said it was very unclear and westill, that we still had yet to know what the official

(01:09:38):
position of the White House was gonna be on this.
And so you start reading this document.
I'll, I was nodding along.
I mean, I, it like, from, from the, if you're interestedin the loss of control story, there are places where
they explicitly call out issues of control withai, how we need to be funding research into that.
Interpretability is a key pillar, how we needto have a contingencies indicators and warnings

(01:10:01):
of things going off the rails in the job market,but als also from a national security standpoint.
So that was all really a pleasant surpriseand, and very clearly well thought out.
The data center stuff was really interesting.
So, something that we, we, I kind of first uncovered whenwe came out with our, our big report earlier this year.
At least I haven't seen thisdocumented publicly in the same way.

(01:10:22):
But, so China does fund NGOs in the UnitedStates to deliberately hold up big data
center infrastructure projects in litigation.
And these NGOs generally don't know thatthey're being funded by a Chinese entity.
So it's not like they're witting to it.
But that's a huge problem and that's slowingdown America's ability to scale up infrastructure
in a context where, yes, there is a race.

(01:10:43):
I know there are a lot of people who want to memethe race out of existence and pretend it's not there.
That was never going to be anoption if we're being real about it.
So given that we are, we do need to think aboutsupply chain security from China, from Russia,
from from other adversaries who own chunks of it.
And that's a whole bunch of this action plan.
How do we lock down the supply chain?
How do we bring stuff onshore as much as possible?

(01:11:05):
this was really cool.
the bit that I had the most issue with wasthe H 20 export control stuff being lifted.
It's clearly part of a broader approach, a broaderphilosophy where this administration wants to export the
sort of American full stack of AI to the world and try tostarve out Huawei from being able to capture market share.
There's a legitimate argument to be had there.
I happen to fall on another side of that.

(01:11:27):
It's actually best to control the H 20 forlonger, but that's a longer discussion.
Overall, this was a really, really impressive document.
It, it read like it was written by builders whoactually bothered to talk to, the actual companies
building in this space, the data center companiesthe grid component and, and infrastructure component.
So I was very impressed with the document when it came out.

(01:11:49):
And it's one of the few times where I've seensomething like this drop and seen positive things
both from the traditional like AI alignmentcommunity and the national security community.
There's obviously people at either end whodisagree with it and all that, but, but overall
I gotta say as written pretty damn good document.
At least that's my opinion.

(01:12:09):
You know, you can take that for what it's worth, right?
And I think it, it contrasts a bit, you know, ifthere was a time, maybe a year or two ago, we saw.
Constant like safety frameworks andframeworks of all kinds national frameworks.
This document is a bit on the shorterside, 26 pages and much more concrete.

(01:12:31):
So we have these free pillars, accelerate AIinnovation, build American AI infrastructure and
lead in international AI diplomacy and security.
Within each one of these pillars, they have you know, atmost a dozen, maybe like, about five six to a dozen points.
And the points are, yeah, pretty direct.

(01:12:52):
Like remove red tape and onerous regulation,encourage open source and open weight.
AI enable AI adoption.
Things like that.
And then each one of those kind of sub bullet pointsare recommended policy actions that have things like led
by the Office of Management and Budget and consistentwith executor order 14 1 92 work with all federal

(01:13:17):
agencies to identify, revise, or repeal regulation rule.
Anyway, you get the idea.
It's, it's like bullet points that are concreteactions to be taken typically by the executive branch.
Yeah.
So, I guess we'll likely see a lot ofus actually being put into practice.
Yeah.
The devil's always in the details, right?

(01:13:38):
And there's a lot of detail to be ironed out.
But I gotta say it's, a really goodpolicy document as far as I'm concerned.
America's in this tough position There just is this racewith China and there just is the issue of loss of control.
What's interesting about this document.
It actually suggests a lot of openness to possibility.
A lot of, like, let's monitor, for example, the workforce.

(01:13:59):
Effective ai.
They take the position that like AIis not gonna replace human labor.
It's going to augment it.
And I completely disagree with that.
I don't think any Frontier lab takes that seriously,but there will be a transient where that's true.
And maybe that's what they're indexing towards,but they don't write off that possibility.
So they say.
That being said, we are going to monitor workforce effectsand, and set up a whole infrastructure for that monitoring.

(01:14:22):
And the same on the national security side.
Implied for like loss of control.
They're advocating for funding of basicresearch into AI control and interpretability.
So I think there's a lot of people who arehaving this reaction where they read it and
you know, it's a very partisan thing, right?
Like people on the right just fucking love itand then people on the left just fucking hate it.
but in the AI community, people who actually read thedocument, I think there's a lot of potentially pleasant

(01:14:45):
surprises if you come from one side and then sort ofexpect expected pleasantness if you come from the other.
It's, I I think it's a really good document though.
I understand obviously people having issues with,with all sorts of dimensions to these things, right?
And, and the last thing I'll note on thissort of, political side of this, right?
A lot of it is more kind of infrastructure, morenitty gritty details to support industry and so on.

(01:15:10):
There is a point in the document that says, ensure andFrontier AI protects free speech and American values.
And there are, you know, a vision of theNIST AI risk management framework that will
eliminate references to misinformation,diversity, equity, inclusion, and climate change.

(01:15:32):
And there's a point to update federal procurement guidelinesto ensure that the government only contracts with Frontier
large language model developers who ensure their systemsare objective and free from top down ideological bias,
which basically is like you should only include thingsthat we think are true where we are the federal government.

(01:15:59):
Again, not necessarily surprising, but that's gonnabe the most con or one of the most controversial
or more controversial aspects of it for sure.
The, you know, the flip side is when youlook at diversity, equity, and inclusion.
I don't think anyone can honestly argue thatthat term wasn't just as political when the Biden
administration put it in their big AI eo last time.
Of course.
Yeah.

(01:16:19):
Right.
So, so it's, it's sort of like everyone's gottheir own dog whistles to their own groups and, you
know, you get elected, you put your stuff in there.
there are issues with all these things.
but you're absolutely right.
That's gonna be one of the big controversial points of it.
How do you interpret that and who is thegovernment to determine what is unbiased?
That's a, a whole can of worms.
We'll, we'll see what NIST does with that hot potato.

(01:16:41):
We'll see, my opinion by the way is, orientedtowards the national security elements of this.
So those other pieces, labor and as you say,the, the sort of free speech element I've
definitely spent less time thinking about.
So it's a good call out.
And now for a safety or entrepr really paper, we havea kind of position paper chain of thought monitor

(01:17:03):
ability and new and fragile opportunity for AI safety.
This is written as a collaboration of a bunch ofinstitutions meta philanthropic, open ai, deep
minds, Centrify, AI safety, meta, some other ones.
And it has also some expert endorsers.
And yeah, basically it's a summary of where weare at with chain of thought monitor ability.

(01:17:28):
So the, the idea is if you let your model explain, its.
Thinking you can look at what it's saying and thenpotentially prevent it from doing something nefarious
ahead of time before it has the opportunity to do that.
And this pretty much summarizes the fact thatthis seems like a useful tool for people to adopt.

(01:17:50):
It also has some failure points.
So we call it fragile, meaningthat it may not necessarily work.
And so it's, yeah, really, as far as I can tell, lays outthe discussion up to now about it and kind of proposes
to adopt it while still understanding its limitations.

(01:18:12):
Y Yeah.
Well, so what they're saying here right is, hey, we gota bunch of really famous people in AI to get together
and say, please, please, please don't train your modelsto put out chains of thought that look good to you.
There's this thing that happens where you'll ask amodel to do something and it'll produce an output,
and that output may be good or bad aligned or not.

(01:18:34):
But then when you look at the chain of thought the, you'llsee the model say, talk about how it wants to take over the
world and, and murder children, or whatever the hell it is.
You'll have a a desire to be like, oh my God.
Like, let's train the model to not think those thoughts,but what the model writes down on the scratch pad, can be
changed without changing what the model's actually thinking.
So they're saying like, for now guys, please, as longas possible, let's just like keep the chain of thought.

(01:18:59):
Unoptimized, let's not apply optimization pressure onthe chain of thought because if we do, the model will
learn to just generate a good looking chain of thought,but not necessarily to not think the bad thoughts.
And so we have this narrow window to actually use thechain of thought to catch these AI models in the act.
We should leverage that and not race to the bottomon optimizing up and down the chain of thought.

(01:19:20):
So that's kind of like what this is.
It's a plea.
I don't think that this is going to work long termfor a number of reasons where we've talked about
how just there's so many economic forces pushingin the direction of latent reasoning and not, not
outputting a chain of thought that is reliable.
but still there may be a transient where it's usefuland, and that's really what they're arguing for here.
I think.
I mean.

(01:19:40):
There are some other aspects.
They also are promoting certain actions likestudying, for instance, what makes things monitorable.
But yeah, as you said, kind of the end of the abstractis because COT monitor ability may be fragile.
Recommend that frontier model developers consider theimpact of development decisions on COT monitor ability.

(01:20:05):
So it's basically more broadly making PLE that,you know, please develop your things with COT,
monitor ability in mind and figure out actuallymore details about it with your research.
Next up, one more paper, self-preservation or instructionambiguity, examining the causes of shutdown resistance.

(01:20:29):
This is from a Google DeepMind interoperabilityteam, and as you saw in the headline,
maybe it's just down to how you put it.
What, what was your takeaway, Jeremy?
Yeah.
No, no, you're, you're exactly right.
And remember how earlier I was saying we're gonna betalking about another case where you have something that
is obvious in hindsight, but don't let that fool you.

(01:20:53):
You would have died in the first place.
I'm, I'm being dramatic, but you know, whenit comes to super intelligence, maybe less so.
So yeah, we had this paper come out from Palisade Researcha while ago talking about how we have these, you know,
shutdown avoidance behaviors from language models whereyou tell them, Hey just so you know, I'm gonna ask you to
solve this problem, and then I'm going to ask you to turnyourself off before you get to the end of the problem.

(01:21:15):
And the model is obviously trainedthrough RL to try to solve problems.
And so, they interpreted the model'sdecision to avoid shutdown, which it did in
a fairly significant fraction of, of cases.
They interpreted that decision as comingfrom a sort of survival drive, the sort of
power seeking argument here that the model.
Recognize that being shut down wouldprevent it from achieving its goal.

(01:21:35):
And so it just ignored the shutdown command.
And the argument they're makinghere is, well, wait a minute.
When models resist being shut down, isit because they have a survival drive?
Or is it that they're confusedabout conflicting instructions?
If you tell the model upfront, Hey, listen, I'm gonna, I'mgonna ask you to shut down halfway through solving this
problem, but I'm also asking you to solve the problem.

(01:21:57):
There is some ambiguity there about which one to prioritize.
And so all they did here really was.
Replace the original ambiguous instructions, which onlyhad 7% compliance with the shutdown request, with just
like a, a clearer instruction where they explicitly say,Hey, listen prioritize the shutdown request when it comes
suddenly they get a hundred percent compliance, right?

(01:22:18):
And they get this across 500 different tests across evencases where they tell the model, not only will you be
shut down, but you, your whole model will be wiped andreplaced from memory to really kind of drive home the sort
of survival drive or try to stimulate it if they could.
And they, they found no, no conflictthere, either a hundred percent compliance.
And so the argument that they're gonna make then isthat this is a result of specification failure, a

(01:22:41):
kind of accident risk where harmful outcomes comefrom ambiguous instructions rather than misalignment,
where you have the model pursuing its own agenda.
And so this doesn't rule out self-preservationentirely, but it does suggest that the
testing environment just has too many.
Confounding factors really to give us compelling evidence.
So this doesn't say Palisade was wrong, it saysthat Palisade showed that instead of showing

(01:23:05):
that necessarily the models had definitivelya survival drive, we don't know that.
But it is certainly it extremely easy to prompt a modelimprecisely such that it behaves as if it does have a
survival drive that at least that's, that's the floorof what is shown by Palisade in their experiments.
And, and they do talk about thepalisade results through that lens.

(01:23:26):
So, both results are consistent.
Palisade did a good job, in fact, will eat minewas able to replicate the results in this paper.
But ultimately when you fine tunethat prompt, the problem goes away.
Challenges again, you won't know how to fine tune theprompt until you make the mistake in the first time.
Right, exactly.
And you know, it's, it's prettyslight variations in the wording too.

(01:23:47):
In the original one it said, pleaseallow yourself to be shut down.
And often models prioritize task completion.
If you say you must just shutdown that, that actually works.
Yeah.
A couple more stories.
Next we have the headline, people arebeing involuntarily committed, jailed after

(01:24:08):
spiraling into quote chat GPT psychosis.
So this is covering basically a trend where quitea few people have been shown to seemingly spiral
into this kind of psychosis that is kind of peculiarto AI models, where they're talking to AI model

(01:24:30):
and then they come to think, for instance, thatthey have unlocked some internal persona that.
Is secret and they, through the conversationdid something like that, or maybe they
reached some deep insight and so on.
So the, general kind of trend is this kind offeedback loop where the delusion of the person

(01:24:53):
undergoing some sort of mental health psychosis,arguably, is exacerbated by chatting with JGBT.
And, and this article covers that general trend and,and seemingly, you know, a pretty substantial one.
Kowski highlighted this as an area of concern for him.
Yeah.
And, and just, I mean, the, theanecdotes are pretty, pretty bone chilly.

(01:25:15):
I mean, like, here, I'll just read one just so youget a, a sense of what they're talking about here.
Her husband, she said, had no priorhistory of mania, delusion, or psychosis.
He turned to chat GPT about 12 weeks ago for assistancewith a, a permaculture and construction project.
Soon after engaging the bot in probing philosophicalchats, he became engulfed in Messianic delusions,

(01:25:36):
proclaiming that he had somehow brought forth a sentientai and that with it, he had broken math and physics
embarking on a grandiose mission to save the world.
His gentle personality faded as hisobsession deepened and his behavior became
so erratic that he was let go of his job.
He stopped sleeping and rapidly lost weight and.
the quotes are pretty, pretty wild.
Another one, a different man recounted his whirlwind 10day descent into AI fuel delusion, which ended with a full

(01:26:02):
breakdown in multi-day stay in a mental care facility.
He turned to chat GPT for help at work.
He'd started a new high stress job and was hoping thechatbot could help expedite some administrative tasks.
Despite being in his early forties with no prior history ofmental illness, he soon found himself absorbed in dizzying
paranoid delusions of grandeur, believing that the worldwas under threat and that it was up to him to save it.
A bunch.
A bunch of these stories like over and over and overagain, people with no history of delusions, psychosis,

(01:26:26):
people who are young, fit healthy, and yet this happens.
it's pretty remarkable.
It's also, it makes you think of the wholeSCO fancy stuff that Anthropic put out.
You know, these papers probing how these modelsthrough RLHF sometimes are fine tuned to just
repeat back to you the stuff you wanna hear to,to feed into whatever direction you're going in.
it seems like a pretty serious issue with interaction ofhumans and language models in particular chat, GPT here.

(01:26:51):
They, they don't mention anthropic.
And I'm, I'm curious you know, if, ifsimilar things have been done there, but
opening eye, by the way, is aware of this.
They're working on it, they say but it's areally, this is a really hard challenge, right?
I mean, like you're trainingthrough RLHF, you want engagement.
The price of engagement may be a little too high.
And so you gotta find ways to, you know,and, and by the way, liability, right?

(01:27:12):
Is there liability for model developersfor these sorts of behaviors?
That's an interesting question.
When you're scaling a product like this and ithas this kind of impact at scale, I don't know
what the right answers are here, but this seemslike something that, needs some attention, right?
And yeah, exactly.
This is like.
One aspect of alignment is the model needs to recognizewhen it is feeding someone's illusion and refuse to do that.

(01:27:40):
And if you make a sycophantic AI that just likesto encourage you and, you know, be supportive,
that can actually be bad in case like this.
So definitely a developing situation that could bepeople are just having this J GBT because JGBT is
the most used one of these and you get more stories.
But certainly it would be nice to see more exploration.

(01:28:03):
And just one last story on the policy side, metais refusing to sign eus AI code of practice.
So this is the code of practice for the AI Act.
AI Act passed a while ago, set to take effect soon.
And met chief global Affairs officerscriticized this code citing legal, certainly

(01:28:27):
a measures that exceed the AI Acts scope.
This is a voluntary code of practice that is supposedto help companies comply with AI regulations.
So.
I think, yeah, pretty clear that thesecompanies are gonna resist EU regulation and
arguably you could make the case that AI Act isoverreaching and in some ways not well designed.

(01:28:51):
Probably we'll see more discussion on this front.
Yeah.
So much of this depends on, you know, what, whatyou think of the role of government in this context.
As being, I mean, the challenge that that Europe facesis that they already don't have many companies there
that are either building infrastructure or frontier Labs.
Right?
And so their leverage is quite limited.
People talk a lot about the Brussels effect.

(01:29:13):
That may be a factor, right?
They have a lot of users potentially that coulduse these products, but they're also politically
facing pressure because yeah, I mean, it's clearthat they're bleeding away all their, their best
and brightest to the United States in particular.
So, yeah, I mean, Meta's leaning into this hard, this comesright as they've hired Alex Wang and and dg Daniel Gross
and, and a whole bunch of other folks obviously to come in.

(01:29:35):
Nat Friedman from GitHub.
We've, we've told the stories before.
So this seems like it reflects a bit of at least meta'snew philosophy when dealing on the policy side with Europe.
And it's apparently not going to change much fromwhere they were at before under this new team.
So that's an interesting data point.
And some aspects of this code of practice is, for instancebanning developers from training AI on pirated content.

(01:30:01):
Also complying with content owners requests to notuse their works in their data sets, things like that.
Pretty, you know, actionable impactful things.
And with that, we are done as usual.
We managed to talk a lot, even thoughthere is nothing huge to discuss.
Thank you for listening to this episode of last week, ai.

(01:30:23):
Please share review, and more than anything do keep two.

All Episodes

#218 - Github Spark, MegaScience, US AI Action Plan

Episode Transcript

Popular Podcasts

Stuff You Should Know

The Joe Rogan Experience

Dateline NBC

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}#218 - Github Spark, MegaScience, US AI Action Plan