September 18, 2026 • 38 mins
Why did a former Anthropic employee’s X post saying AI could kill us all get 170 million views while previous warnings didn’t break through? Reed Albergotti (Semafor), Taylor Lorenz (User Mag), and Nitasha Tiku (The Washington Post) try to make sense of why AI safety talking points became the national conversation, what’s motivating the key voices and whether the debate will lead to effective legislation. Then, could slowing down the AI race actually be good for OpenAI and Anthropic’s bottom lines? And is the frenzy over AI’s existential risk distracting us from the problems AI is already creating?
See omnystudio.com/listener for privacy information.
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:12):
Oh, wait, we've got the queen herself here.
Speaker 2 (00:15):
Natasha, we were worried you'd been killed by AI, but
you're here.
Speaker 1 (00:20):
I said the EAs took you out.
Speaker 3 (00:24):
Or did they? Is this a Natasha deepfake?
Speaker 2 (00:35):
Welcome to Tech Stuff. I'm Asif Al-Hashim, and this is
The Week in Tech, where I'm joined by the world's
most plugged-in reporters... to break down what's really happening in
tech right now. Today, we're joined by Reid Albergotti, tech
editor for Semaphore, Taylor Lorenz of UserMag, and Natasha Tiku,
tech reporter at The Washington Post.
Speaker 3 (00:53):
Welcome all. Thanks for having me.
Speaker 1 (00:55):
Hi. Thanks for having us.
Speaker 2 (00:57):
So we're not dead yet.
Speaker 3 (00:59):
Oh, you know, how would I prove that to you?
I have no idea.
Speaker 2 (01:05):
This has been I mean, I felt I felt like
the in the last week, obviously, everyone's been on the Oh, wow,
AI is going to kill us all train since the
Jacob Coxon viral post. And I feel like I'm my response.
This is so cynical, I can barely bring myself to engage.
But I feel like that's a bad place to be.
And I mean, I have to ask you guys, what's
it what's it been like for you kind of being
(01:26):
in this wave of interest in this beat?
Speaker 3 (01:30):
I find it to be a very uncomfortable situation where like,
as a reporter, I'm actually like, on the other side
of the warnings about, you know, the dangers of some
new technology, right? I'm like, the reporters are supposed to
be the ones warning everyone. And I'm like, no, guys, like,
this is not exactly what you think. I'm like, actually,
(01:50):
now I have anxiety because I need to call my
mom back. She was she had a she was sending
me panicked text messages yesterday about this. So That's where
I'm at.
Speaker 1 (02:03):
I love to hop in the discourse. So I think
this is such a fun thing to talk about. I
am worried about the effect that it's having on people.
I feel like over the past week, I've been hearing
from a lot more people that are like, hey, what
do you know about this stuff? Are we all going
to die? What's happening? And that makes me nervous because
I don't like when the public is scared of technology.
(02:26):
I think it doesn't bode well.
Speaker 3 (02:28):
I would also just say, Oz, that there are a
lot of real concerns from this technology, many of which
we've talked about on this show. And I honestly do
think that when you start talking about human extinction and
this sort of hypothetical future that doesn't exist yet, you're
taking a lot of attention away from stuff that really
does need to be addressed. Fully agree.
Speaker 2 (02:49):
I mean, that's partly why I find I don't want
to engage in the conversation. But why did your mom
call you? What is it that Coxon said that broke
through that Stuart Russell and Jeffrey Hinton and Dario Amadei himself, like,
I genuinely don't fully understand what was different about this
and why.
Speaker 3 (03:07):
It went viral. Well, I think a lot of it's timing,
you know, you, you had had a lot of, uh,
you know, stuff coming out about the hugging face hack.
I think also there's an understanding that these companies are
about to go public now, like there's real money on
the table. And then this guy's resigning and giving up
a lot of money, right? Like he's a young guy.
(03:28):
Like this isn't like a Jeffrey Hinton who's probably made
more money than he's ever going to spend. And it's
like now, you know, retiring anyway. I mean, he's like,
this guy's like saying, I'm giving up my financial future
here to like raise the alarm bells. I think that
does resonate with people. Do you agree, Taylor?
Speaker 1 (03:45):
I was going to also say timing as well. I
think that like it's been building for a while. I
also think that heat perfectly executed his media strategy. And
I think this to the point that people thought it
was like a PSYOP.
Speaker 3 (04:00):
How?
Speaker 2 (04:01):
Explain how.
Speaker 1 (04:02):
I think that we are very familiar with kind of
this process of like sort of what the whistleblower goes
through now. And, you know, Jacob had spoken to the
Wall Street Journal. He had lined up an appearance on
Fox News. He had spoken to other people in the
AI safety communities. There were people ready to like boost
his message very early on. a lot of these AI
safety orgs immediately started reaching out to reporters to amplify
(04:25):
his message and say like, hey, did you see this?
So I feel like all of that also, it was almost,
to me, it reminded me of like a tech launch.
Like I've been talking to a lot of people in
startup world lately and it's like, they'll be like, okay,
you know, we're launching at 11 a.m. At 11.15, you
quote tweet it. Then I send it to the group
chat and these organizations will boost it. And so that's
(04:46):
kind of how it felt like this. And so I
think that that sort of strategic rollout helped it break
through as well.
Speaker 4 (04:52):
He did tell me he sent it to his group chat.
So he said he had a group chat of about
10 people, including Sneha from Encode, which is like a
very strategic and successful AI safety nonprofit. He's also, he
said he was neighbors with Daniel Coccatello, the, you know,
the guy behind AI 2027, that his group house was
(05:15):
next to the driveway of Daniel's group house. And he'd
previously put out a Slack message in the Anthropic Slack.
So they knew it was coming as well. I mean,
it's still hard to get 170 million views on X
these days.
Speaker 2 (05:30):
But what was your what was your conversation with him, Natasha?
Speaker 4 (05:33):
It was very brief. It was 11 minutes. I did
not realize that, or I would not have spent so
much time asking him about how, how did he get
to go so viral? But yeah, I mean, he was,
he didn't have any particular insights to share from his
time at Anthropic. I was like, you know, was there
something that you saw in particular? The only thing that
(05:55):
he mentioned was company Slack, where they're talking about the
end game in a really overt way and the discomfort
of the idea of these executives or a company talking
about the end of the world in this way on
the company Slack. And when I asked him, do you
belong to any particular group? He said, no, even though
(06:16):
he had a lot of friends in AI safety. And
he said, actually, he thought his message went more viral
because The way he talks is not AI safety coded,
which I think is probably true.
Speaker 1 (06:27):
What does that mean? Like, I.
Speaker 4 (06:31):
Mean, it wasn't immediately discernible. I will say he had
I asked him, did you like scrub your presence from
the Internet? Because like even on Nexus, you couldn't find anything.
I'm sure Reid is also a great sleuth and and.
found stuff, but like it was surprisingly little. So normally
if an AI safety person comes out, you know what
scholarships they got, you know, you know, you, you found
(06:51):
them on less wrong. He wasn't out in that way.
But then during an AMA, he kind of said he
was a rationalist.
Speaker 1 (06:56):
I feel like he's absolutely part of that community. But
as Natasha said, he wasn't immediately visible as part of
that community. Um, there's a lot of efforts as well now.
And there's memes about this online of people being like, well,
I'm not completely EA. I'm EA adjacent. I'm not part
of the real whatever. You know, it's like all of
them want to claim that they're not part of this cult.
(07:17):
And fundamentally, they are.
Speaker 4 (07:19):
I'll just say, like, I have even had people who
are on the EA website as an example of an
EA say, like, I'm not really EA. So it's I
can't recall anybody wanting to claim it. Oh, there's some
other stuff that he said. I did also ask him about,
you know, you're getting calls from senators, whatever. What are
your policy goals? And he said he didn't have any
(07:41):
specific ones, but he thought, you know, maybe this would
be really good, good kind of fodder going into Trump's
meetings with Xi. And I asked him, like, oh, did
you time it that way? And he said, no, I
just learned that these talks were happening.
Speaker 1 (07:55):
So ridiculous. Yeah.
Speaker 3 (07:56):
There's a big hope that somehow the U.S. and China
are going to get together. That's been an EA sort
of AI safety hope for a long time.
Speaker 2 (08:05):
I want to unpack the timing a bit more, though,
because the Coxon viral post happened. The whole world became
interested in existential AI risk. And then the CEOs, Altman, Musk,
and Amodei, chose the immediate aftermath of that to do
their call for a slowdown of the frontier.
Speaker 1 (08:25):
Were they.
Speaker 2 (08:26):
consciously adding fuel to the fire? Or were they just
going to do it anyway? And they didn't reflect on
how much the timing of their announcement right after the
Coxon thing will kind of send this to the stratosphere? Like,
what was what was it that they did this week?
And how is it different from last week's news cycle
driven by Coxon, Natasha?
Speaker 4 (08:44):
I mean, I think Dario Amadei has said multiple times
that he is was hoping for some kind of coordination
between you know, for the labs to all agree or,
you know, multiple countries to agree to a slowdown. So
he was motivated. I do know that Sam had been
having these conversations, but I think it's also just like
in a way like a recruiting retention tactic. They know
(09:06):
that their employees care a lot about this and all
of a sudden it doesn't become, you know, anti-capitalist. It's not, um,
you know, it might not be seen in a cynical way.
It looks like there's an opening, you know, and tech
companies are nothing if not like total sheeps in a way.
And so I think that they found cover. And it's
(09:28):
true that a lot of the people who work on
capabilities research are also within this community. And, you know,
you can see a lot of stuff that OpenAI has
done recently to make sure that it's still able to
recruit and retain people from this community. So I think that,
you know, they know that they've been hated. They have
their own reasons why it would behoove them. Like Reid said,
(09:50):
they can't control their models. They don't know what they're
doing right now. Their risk section in the S-1, if
they file to go public, is going to be... the
best sci-fi we've read all year. But can I just
say one thing that I thought was so funny? Dario specifically,
so in his essay, he put this like three-step plan. One,
(10:12):
we have embedded evaluators. Two, You know, the government gives
us an antitrust exemption waiver so that we can coordinate
between the labs and then three China and the U.S. collaborate, whatever.
The fact that he did not anticipate how badly the
embedded evaluators response was going to be because he put
in this company meter, the same one that did the
(10:33):
hugging face report that has like such obvious evidence. EA
connections and that everyone in the industry has been stewing
about the fact that Meter was the one that did
this report. The fact that he didn't even anticipate this
was going to blow up. I'm just like, wait, why
are we listening to him about like global strategy, about geopolitics,
(10:55):
about anything if he couldn't even see that leaving that
off would have saved him and his company a world
of hurt?
Speaker 3 (11:01):
Yeah.
Speaker 2 (11:02):
How did how did Trump react to the call from
Amadei and others?
Speaker 3 (11:07):
Do we? Yeah. I mean, so ridiculous. Like his tweet.
Speaker 1 (11:12):
Trump's posts.
Speaker 4 (11:13):
Yeah.
Speaker 1 (11:13):
Trump's posts have like been really, it's, it's so hard
to tell if they're parody or not. I feel like
sometimes when you read them, but Trump is sort of
going all in on AI. Like, I mean, he, he
also like, called Jensen Wong on stage at the All
In Summit to say that like, you know, he's got
(11:34):
the right approach to AI. He's basically like, he's like
the last hyperscaler maxer left. Like, it's kind of funny.
I mean, people were saying like when Trump dies, there
really is no other pro-AI like politician to that extent.
I mean, he's been talking about how like AI is
the future. People that don't want AI to move forward
(11:55):
are like poor and stupid. And he's fully pro-AI.
Speaker 3 (12:00):
He said all we need is all we need for
AI safety is a genius president or something.
Speaker 4 (12:05):
Yeah.
Speaker 3 (12:06):
Okay.
Speaker 2 (12:07):
Why is he such a mixer? Is it because of
David Saxon is around him or because he believes in
that the keeping the stock market going throughout his presidency
will be good? Or what's his like, what informs his
extreme passion for this?
Speaker 1 (12:20):
I think he genuinely like believes this stuff. I think
his thinking is probably shaped by people like Jensen and
David Sachs. But I think also like he's just got
it in his head that it's good. I mean, this
is a guy that uses AI, you know, he's posting
AI slop or his intern is or whatever. Like, I
think he recognizes that economic power. It is ironically, like,
(12:40):
people have been saying online, like it's like the one
thing that he's kind of able to like cut through
and be quote unquote good on in the sense that like,
I think he is fighting against a lot of like
media pressure and PR spin, but I don't think he
has a coherent regulatory strategy. I think he just sort
of has this vibe that AI is good, good for
(13:00):
the economy, good for his presidency. And so he's going
to be pro AI.
Speaker 3 (13:05):
Well, he gets calls from people and they convince him
and then he tweets, right? There's just, this isn't like
a very, I don't think this is like a very
well thought out thing. And there's a bit of a
rift in the White House between there are some real
kind of like AI safety oriented people now that I
think have some influence. But he probably got a call
from someone, maybe it was Zuckerberg, maybe it was Jensen
(13:27):
over the weekend and decided to tweet that. That's my guess.
Speaker 2 (13:31):
What about Congress? Is Congress united on the like regulate
AI kind of movement now? Are there any kind of,
unleash the brakes figures in the Senate or the House?
Speaker 3 (13:44):
There are tons of bills. There are all these bills, right?
But none of them are going to pass. Like it's,
it's the state that the states that matter, right? The
states will actually pass laws and none of it's actually
going to, even if we, even if they pass all
the laws, let's say Congress could get together and even
pass a law. It's not going to prevent the existential
(14:05):
risk from AI development, right? This is, We make all
these assumptions about technology that hasn't been invented yet that
turn out to be wrong. And I think we will
do the same thing with AI.
Speaker 2 (14:15):
But I want to ask you guys, what are the
actual legitimate existential concerns? I know it's a bit of
an eye roll, but on the other hand, what are
the real scenarios driving existential risk that people are laying
out that are actually somewhat plausible in the 20-year horizon?
Speaker 3 (14:32):
I think the thing that The AI safety people, if
you read their books and all the detailed things they've
written on less wrong, et cetera, the AI always has
to find a way to get around the off switch.
These AI labs, they need everybody to be on their
giant data centers, right? Which is what all the powerful
AI models need right now to run is these giant
(14:52):
data centers, which are also giant off switches. Like the
AI development that could potentially threaten humanity, like isn't even
happening yet. like, we're just talking about making large language
models bigger and bigger. But the real question is, like,
could we one day have AI models that can continuously
learn and can operate, you know, can have human intelligence
(15:14):
with 20 watts of energy? Right? I think these are like,
these are the types of things that actually we need
to pay attention to. And they're sort of coming down
the pike, maybe 1015 years from now. So that's sort
of what I, no one, no one has this sort
of practical conversation when you, when you have, when they
have Coxon on, you know, on, on Anderson Cooper or
(15:35):
whatever these shows, no one actually gets into the details.
And I think that's what, that's partly why everybody's sort
of irrationally just freaked out right now is they're not
actually thinking through like, What actually could destroy humanity, right?
Speaker 4 (15:46):
Why would you ask a 27-year-old who's worked at two
AI companies how something might unfold on a global scale?
They are not the experts, so they're not going to
be able to speak in detail about that.
Speaker 3 (15:59):
Then why are we freaking out because of what they're saying?
Speaker 4 (16:01):
I'm not freaking out.
Speaker 3 (16:03):
Well, we as a society.
Speaker 4 (16:05):
No, no, I know.
Speaker 3 (16:05):
My mom.
Speaker 2 (16:06):
Yeah.
Speaker 1 (16:06):
Yeah.
Speaker 2 (16:13):
When we come back, how might pacing the frontier actually
help AI companies financially?
Speaker 3 (16:20):
Stay with us.
Speaker 2 (16:34):
What's going on with the IPOs? I mean, people are
talking about this being the year of $ 3 trillion IPOs,
but it looks like OpenAI and Anthropic might not happen
this year.
Speaker 3 (16:43):
Or probably won't happen this year. Yeah, because I mean, ultimately, like,
they're not right now, they haven't really figured out their finances.
I think that's pretty clear. I mean, you get you
see a lot of numbers get leaked in the press
about how much revenue these companies have, and they're profitable, etc.
But like, none of this is, you know, audited gap financials, right?
(17:03):
This is all just these are all leaks to the
press to kind of, you know, manipulate investors in some
way or another. So When we actually see the numbers,
and by the way, these companies filed confidentially, right? Like
we haven't seen the official numbers yet. It's not going
to look great. Right now, they're spending billions upon billions
(17:23):
of dollars on these massive training runs, right? That are
making these models better, right? They solve millennium math problems, right?
And that's great. But most businesses, most of these customers
of these companies don't need to solve the Millennium Prize
math problems, right? They need very basic, practical, and reliable
(17:45):
models to automate parts of their workflow. But because of
this race, this competition with these other companies, they have
to keep training these, spend billions of dollars in training
these massive models, which also, by the way, require GPUs
to be crammed into these hundreds of thousands of GPUs
to be crammed into one building because the training runs
(18:07):
have to more or less happen in one place, which
is also not practical from an energy perspective, like getting
enough power or just getting the data centers passed by
some local community. So all of this stuff points to,
we should pace the frontier. And you can call it
what you want, but I mean, it's good for their business.
Speaker 2 (18:28):
Natasha, how much of the pacing the frontier conversation has
been driven by what you've called AI's tendency to hack?
Speaker 4 (18:35):
I mean, I think that the open AI hugging face hack,
which we've talked about a lot, definitely played a big
role in how the public received this and then how
the companies reacted. It's just this really concrete example of
how things could go wrong in very.
In ways that have nothing to do with existential risk
(18:58):
or with, you know, some of the far out ideas
around goal seeking or power hoarding or even sentience. You know,
this is just like, it's also coming out in drips
and drabs. I mean, even just yesterday, there were more,
like I saw like a university from Tennessee was asking
why OpenAI's agents were inside their private system. So I
(19:19):
think that these real world examples, even though we haven't had,
you know, um, actually anybody go after OpenAI for these
hacks yet, like, you know, not the FBI, not a
private company yet. I think that they play a big
role in that. And I think we saw this, too,
in some of the accounts that we're hearing from the
(19:41):
lab employees, like not necessarily Jacob, who hadn't been at
Anthropic that long, but Evan Hubinger, who is a very
well-known AI safety person and has been at Anthropic for
a long time. You know, his kind of quote tweet
that really helped push and legitimize Jacob, I think. He
has been talking for years about the problem that I
(20:04):
had written about recently, which is like, you know, they
hack because we taught them to hack. Like, this is
a problem people identified a decade ago, the way that
we develop these models incentivizes them to be focused on
this one reward, and you can cheat your way to
that reward. I mean, there's been some really interesting reports, too,
(20:26):
just coming from the folks who build some of the
environments where these agents learn to move around and do
the kinds of things humans need them to do and
how sloppy those environments are, how poorly quality tested they are.
And so, you know, it's essentially the humans are hacking,
so then the AI agents are hacking. I mean, I think...
(20:47):
even when people talk about far off problems, when you
sit down and you ask like, but they're doing this
because this is what we told them to do, right?
We told them to operate as a swarm. We told
them to like write down notes as your memory. You know,
we told them to win that exploit bench, the test
that the OpenAI agents were on. I think like, yeah,
(21:11):
it's just coming to the fore that, hey, we might've
known about this problem for a decade, but we have
not at all.
Speaker 1 (21:16):
Figured it out.
Speaker 3 (21:17):
I do think this hacking stuff is an issue, don't you?
I mean, I think it was an issue before AI,
like we weren't doing enough to, you know, shore up
our systems. And it's a bigger issue now.
Speaker 4 (21:27):
You don't need the most advanced models to cause a
ton of chaos. You just need to have, you know,
kind of give them permission or not observe them closely
enough to where they can do this. Like this does
not need to be mythos level agents to break into
some universities.
Speaker 3 (21:45):
Yeah.
Speaker 2 (21:46):
So is there going to be a big investment in
cybersecurity now? Because you don't need any more. Regardless of
how fast the frontier grows, to your point, the cat's
out of the bag. And even open source models probably
are strong enough to do a lot of this type
of work. So do you see a huge boom in
cyber defense investment coming?
Speaker 3 (22:05):
It's not enough. I mean, I was talking with one
of these big defense contractors the other day. And I'm like, well, clearly,
there's going to have to be some huge government effort
to go and shore up all of our you know,
critical infrastructure and make sure it's not running on Windows XP.
And they're like, well, that would just be way too expensive.
And I'm just like, well, guys, like this is going
to be a huge disaster then. I mean, best case
(22:27):
scenario is just it costs the economy a lot of
money because there's just more and more ransomware attacks. Worst
case is they, you know, shut down the electrical electricity
grid like that. That would be horrible. I mean, that's
like basically, you know, worse than a nuclear bomb in
terms of, you know, how many people would die.
Speaker 1 (22:44):
But also cyber attacks are not new. I think, honestly,
we're not listening enough to cybersecurity experts in all of
this conversation. I think they have a lot of nuanced
thoughts on this. I think that they have said many times,
I mean, I agree with Reid that the government is
probably not secure as it should be, but also there
(23:06):
are extremely sophisticated hackers trying to take down our infrastructure
every single day and the government does mitigate those threats.
Speaker 3 (23:16):
Are they really trying to take down the infrastructure every day?
I mean, certainly state actors have like broken into critical infrastructure,
but like, I don't think they actually want to take
it down. Like I, but I, but I think like
if you give that power to anyone, if it's not
just state actors and now it's literally anyone can do it. I,
I do think there's some risk.
Speaker 1 (23:37):
Anyone cannot do it. Anyone cannot do it. Like, we're
talking about like massive, like, I just, this idea that like,
I just, I, I am so skeptical all of it.
And I think that none of the, no cybersecurity people
are the ones out there raising the alarm, going on CNN,
making these cases for like doomsday that like, we all
(23:58):
need to panic right now. What they've said is that we,
we want to make sure that we're prepared. Here are
the steps we could take to be prepared. I think like,
you're not going to see an overnight takedown of the
entire electrical grid. We'll probably see these smaller attacks. Like
once those happen, I think they will raise alarms to
people in power. Like, I just think that like, we're
(24:18):
getting so ahead of ourselves and there's endless amounts of
hypotheticals that you can come up with.
Speaker 3 (24:22):
Oh, I mean, I sort of, I get, I get
what you're saying. Like I, and I, I'm not, I mean,
it makes a lot of sense. I just think like,
and I'm not saying like panic freak out, like we,
the world is about to end, but like, You're right.
There are hackers who are hacking into systems every day
and every big company gets hacked. And taking down hospitals.
Taking down hospitals. There's ransomware attacks that happen every day.
(24:45):
Every company is being hacked every day, for sure. We're
not talking about a step change, but we are talking
about a percentage increase. And what percentage increase is that?
I don't know. Is it 20%? Like there's an economic
cost to that. And there's also just like, I think,
unintended consequences. And I think it would make sense, like
(25:07):
even without AI, it would make sense to do this.
But I think it makes more sense now to have
a real concerted national effort to improve cybersecurity. That's what
I'm saying.
Speaker 1 (25:17):
Sure. I agree with that.
Speaker 2 (25:20):
So you have the kind of people leaving AI labs
and warning of existential risk in one camp. Then you
have the people leading the frontier labs calling to pace
the frontier. Then you have Trump saying, no, don't pace
the frontier. Then you have politicians trying to message test
to figure out if AI will actually bring people to
(25:40):
vote in the midterms and beyond. There's a lot going on.
What happens next? Where is this all going? This seems
really in the eye of a storm right now.
Speaker 3 (25:51):
Probably something else. Sorry, go ahead, Taylor.
Speaker 1 (25:56):
I'm very concerned about where this is going because I
think there's a lot of fear. And to just bring
it back to what Reid was saying right when we
started getting a call from his mom, this has really
broken containment, no pun intended, to normie audiences. And I
think in times of great technological change, which we're definitely in,
you see a lot of fear from the public and
(26:18):
that can often have very negative consequences. I don't think
we get thoughtful, useful policy when the public is scared.
I don't think that a lot of the public will
take advantage of the upsides of this new technology when
they're so scared of it. And so I just, I
think there's a lot of fear. There's a lot of
money in feeding people's fear and denialism. And so I, yeah,
(26:42):
so I'm just very concerned about where that will go.
I don't think it's going to go well. I think
it's going to result in really bad laws and just
a lot of like hurt and pain and needless kind
of angst from from people, especially young people. You know,
a lot of them are feeling like they're growing up
in this world that's falling apart, climate change, et cetera.
Now the, you know, AI is going to kill us all.
(27:03):
Like it's, it's hard for like a, I was talking
to a 16 year old yesterday and it's like, I
get why they feel so nihilistic and upset. Yeah.
Speaker 4 (27:11):
I was going to say class warfare or class warfare
is where we're headed, I think. and also like further
radicalization of billionaires who are extremely resentful about this populist
backlash and this confusion and the fear mongering. And, you know,
and they're already anticipating that this will be a tax,
(27:35):
a huge tax burden for them. And, you know, that
people are be coming out with pitchforks to like take
what's theirs. And, you know, as we've seen for the
past 10 years, like, brainwarmed billionaires are like can be
quite influential in American politics. So I think we'll see
all that. But like one thing I wanted, I do
(27:56):
agree with both Reid and Taylor about this being a terrible,
a terrible perspective to start writing policy. And what Reid
was saying about some of the stuff not coming to
effect for 10 years is I think if we are
legislating about a future that, you know, is completely unknown,
(28:16):
and I think we saw this with social media, even
some of the stuff I wrote about early, that ended
up not being the main problem with social media, right?
And say we just listen to the cybersecurity experts and
do things that help the way that the models are now,
that put in appropriate guardrails or think about the problems
that we're seeing in front of us. We could be
preventing that future, but if we legislate towards that future,
(28:40):
all the stuff we are seeing is just going to fester,
I think.
Speaker 3 (28:45):
Yeah, it's totally unpredictable what will happen if and when
this stuff is invented in 10, 20, 30 years. We
have no idea. So creating legislation now is sure to
have major side effects, unintended consequences that you know, might
create other issues, right? We've seen this before. I mean,
(29:06):
I think a great example, like my colleague Liz Hoffman
wrote about the parallels to the financial crisis. And a
lot of people don't remember, but like there was, and
I think it was 2004, there was like this FBI
agent who was warning about, you know, the mortgage crisis and,
you know, all this fraud in the mortgage industry and
how it was going to create this huge problem in
(29:26):
the financial markets, et cetera. And he just got completely
ignored because everybody was, Completely focused on terrorism at that point, right?
The Patriot Act and all this stuff. And we just
didn't police any of this stuff because of that. And
then we got the financial crisis, which then of course,
led to all of these other issues that we're still
dealing with today in our political system. And, you know,
(29:48):
I think we have to, we should look back and
think about that. Think about like, okay, guys, like, Let's
try to find our center here and our principles that
we really care about and stick to those rather than
go off on hysterical tangents because some 27-year-old said something.
Speaker 2 (30:07):
Is there anything that should be happening on the product
side of AI development that's not happening or that changed
this conversation? Or is what you're all saying, this is
actually the product side has been kind of overemphasized versus.
Speaker 1 (30:23):
I think we really need to invest in open source
AI from America right now. Most of the open weight
models are from China. I think that as we seek
to regulate AI, I really worry about us not discussing
open source software in light of these regulations. There's a
really aggressive attack to, I mean, California just passed a law, Colorado.
(30:47):
There are attacks on open source software development outside of AI.
And I think that going forward, it would be great
if as a country, we recognize the need for open
source software, recognize the need for open weight models, recognize
the need to have transparency and accessibility with any sort
of new technology. And we make sure that any regulation
(31:09):
that we write fosters that type of tech ecosystem. I
worry that the way that a lot of regulation and
a lot of the way we talk about the future
of AI, we're only talking about anthropic and open AI. And,
you know, AI is a lot This is a new
type of technology. It's not only those companies that we're
writing these laws for.
Speaker 3 (31:28):
And Taylor, you brought up liability earlier. I think liability
will shape the product side. These companies will be worried
that they're going to get sued if these agents start
going out and causing economic damage by hacking, etc. But
then you mentioned open source companies. well, who's liable if
I'm running an open source model on my home computer, right?
Speaker 1 (31:51):
This is exactly why we should put the liability, in
my opinion, on the person doing harm. You know, like
the person, you know, if you use any sort of
new technology to commit crimes, then you should be, then
(32:12):
you should be like held liable for those crimes. So like,
if I use, you know, an open source model to
hack a bank, then I hacked a bank. The problem
is me hacking a bank, right?
Speaker 3 (32:22):
Well, here's the thing. I use codecs a lot. Like, so,
and I, and I probably, I have it on a
separate computer and I try to experiment with it and
do all these, all these things, which I think are great. Um, hypothetically,
if I'm trying to like, I don't know, build a,
web app for my family. And for some reason, the
agent decides to go hack into JP Morgan and, you know,
(32:44):
shuts down the financial system in the process. Like, I
don't really think I should be liable. Like I think opening,
I should be, but cause I never asked it to
go hack into JP Morgan.
Speaker 1 (32:54):
But here's the thing. I think that if you put
that liability on the user, the companies will inherently put
in safeguards to prevent that because read it, they would
have no users.
Speaker 3 (33:06):
Well, They have put in safeguards. Right. And nobody's hacking banks.
But that's the second part of my point. I'm not
that worried that my codex is going to go hack
into JP Morgan while it's building my family web app
or whatever, because there are all these safeguards in place, right?
And the hacks that happened with the Hugging Face hack,
(33:29):
et cetera, those were in the lab where they don't
have these same safeguards, right? And so these companies are
already dealing with these liability issues. The product changes are
already being made. And I don't know how big of
a problem, I think Natasha kind of made a similar
point earlier, but I don't know how big of a
problem all these hacks are really because the safeguards basically work.
(33:54):
And I don't think OpenAI or Anthropic are going to
release some harness that is just going to go off
and hack into the banking system inadvertently. I think you're
going to have to do that yourself. And if you
do do that yourself, there's going to be a record
of that. There's going to be a record of you
trying to, I don't know, do whatever it takes, jailbreak
(34:14):
this harness to go off and do what it's not
supposed to do. Or if I'm running an open source
model on my home computer and I go hack into something,
there's going to be a record of that too. And
you're going to be caught. So the product changes, I think,
I think the liability does sort of affect the product
and it is doing its job to some extent. Right.
Speaker 1 (34:35):
But what we don't want, which they're trying to do
in Congress now, is put complete liability on anybody that
makes software. And at a time when software is being
democratized and people want to build their own tools for things,
they shouldn't be subject to these intense regulatory systems that
raise that bar. I think that if you do crimes,
(34:56):
you should be held accountable for those crimes. And if
you didn't intend to do the crime and blah, blah, blah, blah, blah,
you can argue that in court. But as you said, Reid,
it's quite hard to not... These things are not going
off the rails and you are not going to accidentally
hack a bank when you're trying to do something small
with these models. And I think that the people that
are making those arguments are not technically literate. honestly. And so,
(35:19):
you know, I just, I worry.
Speaker 3 (35:20):
We need to differentiate between software and products, right? Like
Codex on my computer is a product, but an open
source model that I download from Hugging Face is not
a product. It's just a piece of software. And I agree.
I don't think anybody should be held liable. I think
the open source ecosystem needs to be allowed to be free.
Can't really stop it anyway, right? Because this stuff can
(35:42):
be made anywhere in the world where they could just
do it outside of the.
Speaker 1 (35:45):
But they can criminalize it.
Speaker 3 (35:47):
They, they can. And I think it would be, I
agree that it would be a really bad idea to
do that. I think, but I do think the products,
you know, there should be product liability. Like, I don't think,
I don't think these companies should, we shouldn't have anything
like a section 230 type of like exemption.
Speaker 1 (36:02):
Well, but when you put product liability on a, on
a system that is primarily, that is, you know, people's
primary way of accessing information, that concerns me a lot
because we've seen the way that these product liability cases
and precedents are being used, and they're not being used
to protect people from cyber attacks. They're being used to
(36:22):
censor access to information. So saying that, like, hey, Chachi
PT is faulty because it gave a teen girl information
about an abortion in Texas.
Speaker 2 (36:31):
Natasha, final thought to you before we wrap?
Speaker 4 (36:34):
I think, like, one good part of this conversation swamp
of debate that we've been mired in for the past
week is the discussion about, I mean, they have been,
the companies would like it to focus on evaluators and
embedded evaluators. I think we should like get back to auditing,
but I'm glad that there is a focus on, hey,
(36:56):
who is even deciding what we should be safety testing?
What are the technical chops of the institutions that are
doing the testing? I think bringing in auditors and people
from different disciplines to look more closely at this, like
that's just a no brainer. And right now it's been
(37:18):
kind of a LARP up until now, I think. And
it would be great to not just bring in cybersecurity people,
but I think that's sort of what Taylor is saying
too with open source is it's like, if you start
building some of these product safety things through open source,
where there's like a huge incentive to do it, it's
going to be better for everyone. Like, yes, let's grow
(37:38):
up and start bringing in some outside experts.
Speaker 2 (37:43):
That's all we have time for today. Thank you all
so much for joining us.
Speaker 1 (37:46):
Thanks for having us.
Speaker 3 (37:47):
Great to be here.
Speaker 2 (38:01):
For Tech Stuff, I'm Osvaldo Schim. This episode was produced
by Eliza Dennis. It was executive produced by me and
Julia Nutter for Kaleidoscope and Katrina Norvell for iHeart Podcasts.
Jack Inslee mixed this episode and Kyle Murdoch wrote our
theme song. A special thank you to Taylor Lorenz, Natasha
Tiku and Reid Albigotti. Please check out all the work
(38:22):
they put out into the world. We're lucky to call
them friends of the pod.
Oh, wait, we've got the queen herself here.
Speaker 2 (00:15):
Natasha, we were worried you'd been killed by AI, but
you're here.
Speaker 1 (00:20):
I said the EAs took you out.
Speaker 3 (00:24):
Or did they? Is this a Natasha deepfake?
Speaker 2 (00:35):
Welcome to Tech Stuff. I'm Asif Al-Hashim, and this is
The Week in Tech, where I'm joined by the world's
most plugged-in reporters... to break down what's really happening in
tech right now. Today, we're joined by Reid Albergotti, tech
editor for Semaphore, Taylor Lorenz of UserMag, and Natasha Tiku,
tech reporter at The Washington Post.
Speaker 3 (00:53):
Welcome all. Thanks for having me.
Speaker 1 (00:55):
Hi. Thanks for having us.
Speaker 2 (00:57):
So we're not dead yet.
Speaker 3 (00:59):
Oh, you know, how would I prove that to you?
I have no idea.
Speaker 2 (01:05):
This has been I mean, I felt I felt like
the in the last week, obviously, everyone's been on the Oh, wow,
AI is going to kill us all train since the
Jacob Coxon viral post. And I feel like I'm my response.
This is so cynical, I can barely bring myself to engage.
But I feel like that's a bad place to be.
And I mean, I have to ask you guys, what's
it what's it been like for you kind of being
(01:26):
in this wave of interest in this beat?
Speaker 3 (01:30):
I find it to be a very uncomfortable situation where like,
as a reporter, I'm actually like, on the other side
of the warnings about, you know, the dangers of some
new technology, right? I'm like, the reporters are supposed to
be the ones warning everyone. And I'm like, no, guys, like,
this is not exactly what you think. I'm like, actually,
(01:50):
now I have anxiety because I need to call my
mom back. She was she had a she was sending
me panicked text messages yesterday about this. So That's where
I'm at.
Speaker 1 (02:03):
I love to hop in the discourse. So I think
this is such a fun thing to talk about. I
am worried about the effect that it's having on people.
I feel like over the past week, I've been hearing
from a lot more people that are like, hey, what
do you know about this stuff? Are we all going
to die? What's happening? And that makes me nervous because
I don't like when the public is scared of technology.
(02:26):
I think it doesn't bode well.
Speaker 3 (02:28):
I would also just say, Oz, that there are a
lot of real concerns from this technology, many of which
we've talked about on this show. And I honestly do
think that when you start talking about human extinction and
this sort of hypothetical future that doesn't exist yet, you're
taking a lot of attention away from stuff that really
does need to be addressed. Fully agree.
Speaker 2 (02:49):
I mean, that's partly why I find I don't want
to engage in the conversation. But why did your mom
call you? What is it that Coxon said that broke
through that Stuart Russell and Jeffrey Hinton and Dario Amadei himself, like,
I genuinely don't fully understand what was different about this
and why.
Speaker 3 (03:07):
It went viral. Well, I think a lot of it's timing,
you know, you, you had had a lot of, uh,
you know, stuff coming out about the hugging face hack.
I think also there's an understanding that these companies are
about to go public now, like there's real money on
the table. And then this guy's resigning and giving up
a lot of money, right? Like he's a young guy.
(03:28):
Like this isn't like a Jeffrey Hinton who's probably made
more money than he's ever going to spend. And it's
like now, you know, retiring anyway. I mean, he's like,
this guy's like saying, I'm giving up my financial future
here to like raise the alarm bells. I think that
does resonate with people. Do you agree, Taylor?
Speaker 1 (03:45):
I was going to also say timing as well. I
think that like it's been building for a while. I
also think that heat perfectly executed his media strategy. And
I think this to the point that people thought it
was like a PSYOP.
Speaker 3 (04:00):
How?
Speaker 2 (04:01):
Explain how.
Speaker 1 (04:02):
I think that we are very familiar with kind of
this process of like sort of what the whistleblower goes
through now. And, you know, Jacob had spoken to the
Wall Street Journal. He had lined up an appearance on
Fox News. He had spoken to other people in the
AI safety communities. There were people ready to like boost
his message very early on. a lot of these AI
safety orgs immediately started reaching out to reporters to amplify
(04:25):
his message and say like, hey, did you see this?
So I feel like all of that also, it was almost,
to me, it reminded me of like a tech launch.
Like I've been talking to a lot of people in
startup world lately and it's like, they'll be like, okay,
you know, we're launching at 11 a.m. At 11.15, you
quote tweet it. Then I send it to the group
chat and these organizations will boost it. And so that's
(04:46):
kind of how it felt like this. And so I
think that that sort of strategic rollout helped it break
through as well.
Speaker 4 (04:52):
He did tell me he sent it to his group chat.
So he said he had a group chat of about
10 people, including Sneha from Encode, which is like a
very strategic and successful AI safety nonprofit. He's also, he
said he was neighbors with Daniel Coccatello, the, you know,
the guy behind AI 2027, that his group house was
(05:15):
next to the driveway of Daniel's group house. And he'd
previously put out a Slack message in the Anthropic Slack.
So they knew it was coming as well. I mean,
it's still hard to get 170 million views on X
these days.
Speaker 2 (05:30):
But what was your what was your conversation with him, Natasha?
Speaker 4 (05:33):
It was very brief. It was 11 minutes. I did
not realize that, or I would not have spent so
much time asking him about how, how did he get
to go so viral? But yeah, I mean, he was,
he didn't have any particular insights to share from his
time at Anthropic. I was like, you know, was there
something that you saw in particular? The only thing that
(05:55):
he mentioned was company Slack, where they're talking about the
end game in a really overt way and the discomfort
of the idea of these executives or a company talking
about the end of the world in this way on
the company Slack. And when I asked him, do you
belong to any particular group? He said, no, even though
(06:16):
he had a lot of friends in AI safety. And
he said, actually, he thought his message went more viral
because The way he talks is not AI safety coded,
which I think is probably true.
Speaker 1 (06:27):
What does that mean? Like, I.
Speaker 4 (06:31):
Mean, it wasn't immediately discernible. I will say he had
I asked him, did you like scrub your presence from
the Internet? Because like even on Nexus, you couldn't find anything.
I'm sure Reid is also a great sleuth and and.
found stuff, but like it was surprisingly little. So normally
if an AI safety person comes out, you know what
scholarships they got, you know, you know, you, you found
(06:51):
them on less wrong. He wasn't out in that way.
But then during an AMA, he kind of said he
was a rationalist.
Speaker 1 (06:56):
I feel like he's absolutely part of that community. But
as Natasha said, he wasn't immediately visible as part of
that community. Um, there's a lot of efforts as well now.
And there's memes about this online of people being like, well,
I'm not completely EA. I'm EA adjacent. I'm not part
of the real whatever. You know, it's like all of
them want to claim that they're not part of this cult.
(07:17):
And fundamentally, they are.
Speaker 4 (07:19):
I'll just say, like, I have even had people who
are on the EA website as an example of an
EA say, like, I'm not really EA. So it's I
can't recall anybody wanting to claim it. Oh, there's some
other stuff that he said. I did also ask him about,
you know, you're getting calls from senators, whatever. What are
your policy goals? And he said he didn't have any
(07:41):
specific ones, but he thought, you know, maybe this would
be really good, good kind of fodder going into Trump's
meetings with Xi. And I asked him, like, oh, did
you time it that way? And he said, no, I
just learned that these talks were happening.
Speaker 1 (07:55):
So ridiculous. Yeah.
Speaker 3 (07:56):
There's a big hope that somehow the U.S. and China
are going to get together. That's been an EA sort
of AI safety hope for a long time.
Speaker 2 (08:05):
I want to unpack the timing a bit more, though,
because the Coxon viral post happened. The whole world became
interested in existential AI risk. And then the CEOs, Altman, Musk,
and Amodei, chose the immediate aftermath of that to do
their call for a slowdown of the frontier.
Speaker 1 (08:25):
Were they.
Speaker 2 (08:26):
consciously adding fuel to the fire? Or were they just
going to do it anyway? And they didn't reflect on
how much the timing of their announcement right after the
Coxon thing will kind of send this to the stratosphere? Like,
what was what was it that they did this week?
And how is it different from last week's news cycle
driven by Coxon, Natasha?
Speaker 4 (08:44):
I mean, I think Dario Amadei has said multiple times
that he is was hoping for some kind of coordination
between you know, for the labs to all agree or,
you know, multiple countries to agree to a slowdown. So
he was motivated. I do know that Sam had been
having these conversations, but I think it's also just like
in a way like a recruiting retention tactic. They know
(09:06):
that their employees care a lot about this and all
of a sudden it doesn't become, you know, anti-capitalist. It's not, um,
you know, it might not be seen in a cynical way.
It looks like there's an opening, you know, and tech
companies are nothing if not like total sheeps in a way.
And so I think that they found cover. And it's
(09:28):
true that a lot of the people who work on
capabilities research are also within this community. And, you know,
you can see a lot of stuff that OpenAI has
done recently to make sure that it's still able to
recruit and retain people from this community. So I think that,
you know, they know that they've been hated. They have
their own reasons why it would behoove them. Like Reid said,
(09:50):
they can't control their models. They don't know what they're
doing right now. Their risk section in the S-1, if
they file to go public, is going to be... the
best sci-fi we've read all year. But can I just
say one thing that I thought was so funny? Dario specifically,
so in his essay, he put this like three-step plan. One,
(10:12):
we have embedded evaluators. Two, You know, the government gives
us an antitrust exemption waiver so that we can coordinate
between the labs and then three China and the U.S. collaborate, whatever.
The fact that he did not anticipate how badly the
embedded evaluators response was going to be because he put
in this company meter, the same one that did the
(10:33):
hugging face report that has like such obvious evidence. EA
connections and that everyone in the industry has been stewing
about the fact that Meter was the one that did
this report. The fact that he didn't even anticipate this
was going to blow up. I'm just like, wait, why
are we listening to him about like global strategy, about geopolitics,
(10:55):
about anything if he couldn't even see that leaving that
off would have saved him and his company a world
of hurt?
Speaker 3 (11:01):
Yeah.
Speaker 2 (11:02):
How did how did Trump react to the call from
Amadei and others?
Speaker 3 (11:07):
Do we? Yeah. I mean, so ridiculous. Like his tweet.
Speaker 1 (11:12):
Trump's posts.
Speaker 4 (11:13):
Yeah.
Speaker 1 (11:13):
Trump's posts have like been really, it's, it's so hard
to tell if they're parody or not. I feel like
sometimes when you read them, but Trump is sort of
going all in on AI. Like, I mean, he, he
also like, called Jensen Wong on stage at the All
In Summit to say that like, you know, he's got
(11:34):
the right approach to AI. He's basically like, he's like
the last hyperscaler maxer left. Like, it's kind of funny.
I mean, people were saying like when Trump dies, there
really is no other pro-AI like politician to that extent.
I mean, he's been talking about how like AI is
the future. People that don't want AI to move forward
(11:55):
are like poor and stupid. And he's fully pro-AI.
Speaker 3 (12:00):
He said all we need is all we need for
AI safety is a genius president or something.
Speaker 4 (12:05):
Yeah.
Speaker 3 (12:06):
Okay.
Speaker 2 (12:07):
Why is he such a mixer? Is it because of
David Saxon is around him or because he believes in
that the keeping the stock market going throughout his presidency
will be good? Or what's his like, what informs his
extreme passion for this?
Speaker 1 (12:20):
I think he genuinely like believes this stuff. I think
his thinking is probably shaped by people like Jensen and
David Sachs. But I think also like he's just got
it in his head that it's good. I mean, this
is a guy that uses AI, you know, he's posting
AI slop or his intern is or whatever. Like, I
think he recognizes that economic power. It is ironically, like,
(12:40):
people have been saying online, like it's like the one
thing that he's kind of able to like cut through
and be quote unquote good on in the sense that like,
I think he is fighting against a lot of like
media pressure and PR spin, but I don't think he
has a coherent regulatory strategy. I think he just sort
of has this vibe that AI is good, good for
(13:00):
the economy, good for his presidency. And so he's going
to be pro AI.
Speaker 3 (13:05):
Well, he gets calls from people and they convince him
and then he tweets, right? There's just, this isn't like
a very, I don't think this is like a very
well thought out thing. And there's a bit of a
rift in the White House between there are some real
kind of like AI safety oriented people now that I
think have some influence. But he probably got a call
from someone, maybe it was Zuckerberg, maybe it was Jensen
(13:27):
over the weekend and decided to tweet that. That's my guess.
Speaker 2 (13:31):
What about Congress? Is Congress united on the like regulate
AI kind of movement now? Are there any kind of,
unleash the brakes figures in the Senate or the House?
Speaker 3 (13:44):
There are tons of bills. There are all these bills, right?
But none of them are going to pass. Like it's,
it's the state that the states that matter, right? The
states will actually pass laws and none of it's actually
going to, even if we, even if they pass all
the laws, let's say Congress could get together and even
pass a law. It's not going to prevent the existential
(14:05):
risk from AI development, right? This is, We make all
these assumptions about technology that hasn't been invented yet that
turn out to be wrong. And I think we will
do the same thing with AI.
Speaker 2 (14:15):
But I want to ask you guys, what are the
actual legitimate existential concerns? I know it's a bit of
an eye roll, but on the other hand, what are
the real scenarios driving existential risk that people are laying
out that are actually somewhat plausible in the 20-year horizon?
Speaker 3 (14:32):
I think the thing that The AI safety people, if
you read their books and all the detailed things they've
written on less wrong, et cetera, the AI always has
to find a way to get around the off switch.
These AI labs, they need everybody to be on their
giant data centers, right? Which is what all the powerful
AI models need right now to run is these giant
(14:52):
data centers, which are also giant off switches. Like the
AI development that could potentially threaten humanity, like isn't even
happening yet. like, we're just talking about making large language
models bigger and bigger. But the real question is, like,
could we one day have AI models that can continuously
learn and can operate, you know, can have human intelligence
(15:14):
with 20 watts of energy? Right? I think these are like,
these are the types of things that actually we need
to pay attention to. And they're sort of coming down
the pike, maybe 1015 years from now. So that's sort
of what I, no one, no one has this sort
of practical conversation when you, when you have, when they
have Coxon on, you know, on, on Anderson Cooper or
(15:35):
whatever these shows, no one actually gets into the details.
And I think that's what, that's partly why everybody's sort
of irrationally just freaked out right now is they're not
actually thinking through like, What actually could destroy humanity, right?
Speaker 4 (15:46):
Why would you ask a 27-year-old who's worked at two
AI companies how something might unfold on a global scale?
They are not the experts, so they're not going to
be able to speak in detail about that.
Speaker 3 (15:59):
Then why are we freaking out because of what they're saying?
Speaker 4 (16:01):
I'm not freaking out.
Speaker 3 (16:03):
Well, we as a society.
Speaker 4 (16:05):
No, no, I know.
Speaker 3 (16:05):
My mom.
Speaker 2 (16:06):
Yeah.
Speaker 1 (16:06):
Yeah.
Speaker 2 (16:13):
When we come back, how might pacing the frontier actually
help AI companies financially?
Speaker 3 (16:20):
Stay with us.
Speaker 2 (16:34):
What's going on with the IPOs? I mean, people are
talking about this being the year of $ 3 trillion IPOs,
but it looks like OpenAI and Anthropic might not happen
this year.
Speaker 3 (16:43):
Or probably won't happen this year. Yeah, because I mean, ultimately, like,
they're not right now, they haven't really figured out their finances.
I think that's pretty clear. I mean, you get you
see a lot of numbers get leaked in the press
about how much revenue these companies have, and they're profitable, etc.
But like, none of this is, you know, audited gap financials, right?
(17:03):
This is all just these are all leaks to the
press to kind of, you know, manipulate investors in some
way or another. So When we actually see the numbers,
and by the way, these companies filed confidentially, right? Like
we haven't seen the official numbers yet. It's not going
to look great. Right now, they're spending billions upon billions
(17:23):
of dollars on these massive training runs, right? That are
making these models better, right? They solve millennium math problems, right?
And that's great. But most businesses, most of these customers
of these companies don't need to solve the Millennium Prize
math problems, right? They need very basic, practical, and reliable
(17:45):
models to automate parts of their workflow. But because of
this race, this competition with these other companies, they have
to keep training these, spend billions of dollars in training
these massive models, which also, by the way, require GPUs
to be crammed into these hundreds of thousands of GPUs
to be crammed into one building because the training runs
(18:07):
have to more or less happen in one place, which
is also not practical from an energy perspective, like getting
enough power or just getting the data centers passed by
some local community. So all of this stuff points to,
we should pace the frontier. And you can call it
what you want, but I mean, it's good for their business.
Speaker 2 (18:28):
Natasha, how much of the pacing the frontier conversation has
been driven by what you've called AI's tendency to hack?
Speaker 4 (18:35):
I mean, I think that the open AI hugging face hack,
which we've talked about a lot, definitely played a big
role in how the public received this and then how
the companies reacted. It's just this really concrete example of
how things could go wrong in very.
In ways that have nothing to do with existential risk
(18:58):
or with, you know, some of the far out ideas
around goal seeking or power hoarding or even sentience. You know,
this is just like, it's also coming out in drips
and drabs. I mean, even just yesterday, there were more,
like I saw like a university from Tennessee was asking
why OpenAI's agents were inside their private system. So I
(19:19):
think that these real world examples, even though we haven't had,
you know, um, actually anybody go after OpenAI for these
hacks yet, like, you know, not the FBI, not a
private company yet. I think that they play a big
role in that. And I think we saw this, too,
in some of the accounts that we're hearing from the
(19:41):
lab employees, like not necessarily Jacob, who hadn't been at
Anthropic that long, but Evan Hubinger, who is a very
well-known AI safety person and has been at Anthropic for
a long time. You know, his kind of quote tweet
that really helped push and legitimize Jacob, I think. He
has been talking for years about the problem that I
(20:04):
had written about recently, which is like, you know, they
hack because we taught them to hack. Like, this is
a problem people identified a decade ago, the way that
we develop these models incentivizes them to be focused on
this one reward, and you can cheat your way to
that reward. I mean, there's been some really interesting reports, too,
(20:26):
just coming from the folks who build some of the
environments where these agents learn to move around and do
the kinds of things humans need them to do and
how sloppy those environments are, how poorly quality tested they are.
And so, you know, it's essentially the humans are hacking,
so then the AI agents are hacking. I mean, I think...
(20:47):
even when people talk about far off problems, when you
sit down and you ask like, but they're doing this
because this is what we told them to do, right?
We told them to operate as a swarm. We told
them to like write down notes as your memory. You know,
we told them to win that exploit bench, the test
that the OpenAI agents were on. I think like, yeah,
(21:11):
it's just coming to the fore that, hey, we might've
known about this problem for a decade, but we have
not at all.
Speaker 1 (21:16):
Figured it out.
Speaker 3 (21:17):
I do think this hacking stuff is an issue, don't you?
I mean, I think it was an issue before AI,
like we weren't doing enough to, you know, shore up
our systems. And it's a bigger issue now.
Speaker 4 (21:27):
You don't need the most advanced models to cause a
ton of chaos. You just need to have, you know,
kind of give them permission or not observe them closely
enough to where they can do this. Like this does
not need to be mythos level agents to break into
some universities.
Speaker 3 (21:45):
Yeah.
Speaker 2 (21:46):
So is there going to be a big investment in
cybersecurity now? Because you don't need any more. Regardless of
how fast the frontier grows, to your point, the cat's
out of the bag. And even open source models probably
are strong enough to do a lot of this type
of work. So do you see a huge boom in
cyber defense investment coming?
Speaker 3 (22:05):
It's not enough. I mean, I was talking with one
of these big defense contractors the other day. And I'm like, well, clearly,
there's going to have to be some huge government effort
to go and shore up all of our you know,
critical infrastructure and make sure it's not running on Windows XP.
And they're like, well, that would just be way too expensive.
And I'm just like, well, guys, like this is going
to be a huge disaster then. I mean, best case
(22:27):
scenario is just it costs the economy a lot of
money because there's just more and more ransomware attacks. Worst
case is they, you know, shut down the electrical electricity
grid like that. That would be horrible. I mean, that's
like basically, you know, worse than a nuclear bomb in
terms of, you know, how many people would die.
Speaker 1 (22:44):
But also cyber attacks are not new. I think, honestly,
we're not listening enough to cybersecurity experts in all of
this conversation. I think they have a lot of nuanced
thoughts on this. I think that they have said many times,
I mean, I agree with Reid that the government is
probably not secure as it should be, but also there
(23:06):
are extremely sophisticated hackers trying to take down our infrastructure
every single day and the government does mitigate those threats.
Speaker 3 (23:16):
Are they really trying to take down the infrastructure every day?
I mean, certainly state actors have like broken into critical infrastructure,
but like, I don't think they actually want to take
it down. Like I, but I, but I think like
if you give that power to anyone, if it's not
just state actors and now it's literally anyone can do it. I,
I do think there's some risk.
Speaker 1 (23:37):
Anyone cannot do it. Anyone cannot do it. Like, we're
talking about like massive, like, I just, this idea that like,
I just, I, I am so skeptical all of it.
And I think that none of the, no cybersecurity people
are the ones out there raising the alarm, going on CNN,
making these cases for like doomsday that like, we all
(23:58):
need to panic right now. What they've said is that we,
we want to make sure that we're prepared. Here are
the steps we could take to be prepared. I think like,
you're not going to see an overnight takedown of the
entire electrical grid. We'll probably see these smaller attacks. Like
once those happen, I think they will raise alarms to
people in power. Like, I just think that like, we're
(24:18):
getting so ahead of ourselves and there's endless amounts of
hypotheticals that you can come up with.
Speaker 3 (24:22):
Oh, I mean, I sort of, I get, I get
what you're saying. Like I, and I, I'm not, I mean,
it makes a lot of sense. I just think like,
and I'm not saying like panic freak out, like we,
the world is about to end, but like, You're right.
There are hackers who are hacking into systems every day
and every big company gets hacked. And taking down hospitals.
Taking down hospitals. There's ransomware attacks that happen every day.
(24:45):
Every company is being hacked every day, for sure. We're
not talking about a step change, but we are talking
about a percentage increase. And what percentage increase is that?
I don't know. Is it 20%? Like there's an economic
cost to that. And there's also just like, I think,
unintended consequences. And I think it would make sense, like
(25:07):
even without AI, it would make sense to do this.
But I think it makes more sense now to have
a real concerted national effort to improve cybersecurity. That's what
I'm saying.
Speaker 1 (25:17):
Sure. I agree with that.
Speaker 2 (25:20):
So you have the kind of people leaving AI labs
and warning of existential risk in one camp. Then you
have the people leading the frontier labs calling to pace
the frontier. Then you have Trump saying, no, don't pace
the frontier. Then you have politicians trying to message test
to figure out if AI will actually bring people to
(25:40):
vote in the midterms and beyond. There's a lot going on.
What happens next? Where is this all going? This seems
really in the eye of a storm right now.
Speaker 3 (25:51):
Probably something else. Sorry, go ahead, Taylor.
Speaker 1 (25:56):
I'm very concerned about where this is going because I
think there's a lot of fear. And to just bring
it back to what Reid was saying right when we
started getting a call from his mom, this has really
broken containment, no pun intended, to normie audiences. And I
think in times of great technological change, which we're definitely in,
you see a lot of fear from the public and
(26:18):
that can often have very negative consequences. I don't think
we get thoughtful, useful policy when the public is scared.
I don't think that a lot of the public will
take advantage of the upsides of this new technology when
they're so scared of it. And so I just, I
think there's a lot of fear. There's a lot of
money in feeding people's fear and denialism. And so I, yeah,
(26:42):
so I'm just very concerned about where that will go.
I don't think it's going to go well. I think
it's going to result in really bad laws and just
a lot of like hurt and pain and needless kind
of angst from from people, especially young people. You know,
a lot of them are feeling like they're growing up
in this world that's falling apart, climate change, et cetera.
Now the, you know, AI is going to kill us all.
(27:03):
Like it's, it's hard for like a, I was talking
to a 16 year old yesterday and it's like, I
get why they feel so nihilistic and upset. Yeah.
Speaker 4 (27:11):
I was going to say class warfare or class warfare
is where we're headed, I think. and also like further
radicalization of billionaires who are extremely resentful about this populist
backlash and this confusion and the fear mongering. And, you know,
and they're already anticipating that this will be a tax,
(27:35):
a huge tax burden for them. And, you know, that
people are be coming out with pitchforks to like take
what's theirs. And, you know, as we've seen for the
past 10 years, like, brainwarmed billionaires are like can be
quite influential in American politics. So I think we'll see
all that. But like one thing I wanted, I do
(27:56):
agree with both Reid and Taylor about this being a terrible,
a terrible perspective to start writing policy. And what Reid
was saying about some of the stuff not coming to
effect for 10 years is I think if we are
legislating about a future that, you know, is completely unknown,
(28:16):
and I think we saw this with social media, even
some of the stuff I wrote about early, that ended
up not being the main problem with social media, right?
And say we just listen to the cybersecurity experts and
do things that help the way that the models are now,
that put in appropriate guardrails or think about the problems
that we're seeing in front of us. We could be
preventing that future, but if we legislate towards that future,
(28:40):
all the stuff we are seeing is just going to fester,
I think.
Speaker 3 (28:45):
Yeah, it's totally unpredictable what will happen if and when
this stuff is invented in 10, 20, 30 years. We
have no idea. So creating legislation now is sure to
have major side effects, unintended consequences that you know, might
create other issues, right? We've seen this before. I mean,
(29:06):
I think a great example, like my colleague Liz Hoffman
wrote about the parallels to the financial crisis. And a
lot of people don't remember, but like there was, and
I think it was 2004, there was like this FBI
agent who was warning about, you know, the mortgage crisis and,
you know, all this fraud in the mortgage industry and
how it was going to create this huge problem in
(29:26):
the financial markets, et cetera. And he just got completely
ignored because everybody was, Completely focused on terrorism at that point, right?
The Patriot Act and all this stuff. And we just
didn't police any of this stuff because of that. And
then we got the financial crisis, which then of course,
led to all of these other issues that we're still
dealing with today in our political system. And, you know,
(29:48):
I think we have to, we should look back and
think about that. Think about like, okay, guys, like, Let's
try to find our center here and our principles that
we really care about and stick to those rather than
go off on hysterical tangents because some 27-year-old said something.
Speaker 2 (30:07):
Is there anything that should be happening on the product
side of AI development that's not happening or that changed
this conversation? Or is what you're all saying, this is
actually the product side has been kind of overemphasized versus.
Speaker 1 (30:23):
I think we really need to invest in open source
AI from America right now. Most of the open weight
models are from China. I think that as we seek
to regulate AI, I really worry about us not discussing
open source software in light of these regulations. There's a
really aggressive attack to, I mean, California just passed a law, Colorado.
(30:47):
There are attacks on open source software development outside of AI.
And I think that going forward, it would be great
if as a country, we recognize the need for open
source software, recognize the need for open weight models, recognize
the need to have transparency and accessibility with any sort
of new technology. And we make sure that any regulation
(31:09):
that we write fosters that type of tech ecosystem. I
worry that the way that a lot of regulation and
a lot of the way we talk about the future
of AI, we're only talking about anthropic and open AI. And,
you know, AI is a lot This is a new
type of technology. It's not only those companies that we're
writing these laws for.
Speaker 3 (31:28):
And Taylor, you brought up liability earlier. I think liability
will shape the product side. These companies will be worried
that they're going to get sued if these agents start
going out and causing economic damage by hacking, etc. But
then you mentioned open source companies. well, who's liable if
I'm running an open source model on my home computer, right?
Speaker 1 (31:51):
This is exactly why we should put the liability, in
my opinion, on the person doing harm. You know, like
the person, you know, if you use any sort of
new technology to commit crimes, then you should be, then
(32:12):
you should be like held liable for those crimes. So like,
if I use, you know, an open source model to
hack a bank, then I hacked a bank. The problem
is me hacking a bank, right?
Speaker 3 (32:22):
Well, here's the thing. I use codecs a lot. Like, so,
and I, and I probably, I have it on a
separate computer and I try to experiment with it and
do all these, all these things, which I think are great. Um, hypothetically,
if I'm trying to like, I don't know, build a,
web app for my family. And for some reason, the
agent decides to go hack into JP Morgan and, you know,
(32:44):
shuts down the financial system in the process. Like, I
don't really think I should be liable. Like I think opening,
I should be, but cause I never asked it to
go hack into JP Morgan.
Speaker 1 (32:54):
But here's the thing. I think that if you put
that liability on the user, the companies will inherently put
in safeguards to prevent that because read it, they would
have no users.
Speaker 3 (33:06):
Well, They have put in safeguards. Right. And nobody's hacking banks.
But that's the second part of my point. I'm not
that worried that my codex is going to go hack
into JP Morgan while it's building my family web app
or whatever, because there are all these safeguards in place, right?
And the hacks that happened with the Hugging Face hack,
(33:29):
et cetera, those were in the lab where they don't
have these same safeguards, right? And so these companies are
already dealing with these liability issues. The product changes are
already being made. And I don't know how big of
a problem, I think Natasha kind of made a similar
point earlier, but I don't know how big of a
problem all these hacks are really because the safeguards basically work.
(33:54):
And I don't think OpenAI or Anthropic are going to
release some harness that is just going to go off
and hack into the banking system inadvertently. I think you're
going to have to do that yourself. And if you
do do that yourself, there's going to be a record
of that. There's going to be a record of you
trying to, I don't know, do whatever it takes, jailbreak
(34:14):
this harness to go off and do what it's not
supposed to do. Or if I'm running an open source
model on my home computer and I go hack into something,
there's going to be a record of that too. And
you're going to be caught. So the product changes, I think,
I think the liability does sort of affect the product
and it is doing its job to some extent. Right.
Speaker 1 (34:35):
But what we don't want, which they're trying to do
in Congress now, is put complete liability on anybody that
makes software. And at a time when software is being
democratized and people want to build their own tools for things,
they shouldn't be subject to these intense regulatory systems that
raise that bar. I think that if you do crimes,
(34:56):
you should be held accountable for those crimes. And if
you didn't intend to do the crime and blah, blah, blah, blah, blah,
you can argue that in court. But as you said, Reid,
it's quite hard to not... These things are not going
off the rails and you are not going to accidentally
hack a bank when you're trying to do something small
with these models. And I think that the people that
are making those arguments are not technically literate. honestly. And so,
(35:19):
you know, I just, I worry.
Speaker 3 (35:20):
We need to differentiate between software and products, right? Like
Codex on my computer is a product, but an open
source model that I download from Hugging Face is not
a product. It's just a piece of software. And I agree.
I don't think anybody should be held liable. I think
the open source ecosystem needs to be allowed to be free.
Can't really stop it anyway, right? Because this stuff can
(35:42):
be made anywhere in the world where they could just
do it outside of the.
Speaker 1 (35:45):
But they can criminalize it.
Speaker 3 (35:47):
They, they can. And I think it would be, I
agree that it would be a really bad idea to
do that. I think, but I do think the products,
you know, there should be product liability. Like, I don't think,
I don't think these companies should, we shouldn't have anything
like a section 230 type of like exemption.
Speaker 1 (36:02):
Well, but when you put product liability on a, on
a system that is primarily, that is, you know, people's
primary way of accessing information, that concerns me a lot
because we've seen the way that these product liability cases
and precedents are being used, and they're not being used
to protect people from cyber attacks. They're being used to
(36:22):
censor access to information. So saying that, like, hey, Chachi
PT is faulty because it gave a teen girl information
about an abortion in Texas.
Speaker 2 (36:31):
Natasha, final thought to you before we wrap?
Speaker 4 (36:34):
I think, like, one good part of this conversation swamp
of debate that we've been mired in for the past
week is the discussion about, I mean, they have been,
the companies would like it to focus on evaluators and
embedded evaluators. I think we should like get back to auditing,
but I'm glad that there is a focus on, hey,
(36:56):
who is even deciding what we should be safety testing?
What are the technical chops of the institutions that are
doing the testing? I think bringing in auditors and people
from different disciplines to look more closely at this, like
that's just a no brainer. And right now it's been
(37:18):
kind of a LARP up until now, I think. And
it would be great to not just bring in cybersecurity people,
but I think that's sort of what Taylor is saying
too with open source is it's like, if you start
building some of these product safety things through open source,
where there's like a huge incentive to do it, it's
going to be better for everyone. Like, yes, let's grow
(37:38):
up and start bringing in some outside experts.
Speaker 2 (37:43):
That's all we have time for today. Thank you all
so much for joining us.
Speaker 1 (37:46):
Thanks for having us.
Speaker 3 (37:47):
Great to be here.
Speaker 2 (38:01):
For Tech Stuff, I'm Osvaldo Schim. This episode was produced
by Eliza Dennis. It was executive produced by me and
Julia Nutter for Kaleidoscope and Katrina Norvell for iHeart Podcasts.
Jack Inslee mixed this episode and Kyle Murdoch wrote our
theme song. A special thank you to Taylor Lorenz, Natasha
Tiku and Reid Albigotti. Please check out all the work
(38:22):
they put out into the world. We're lucky to call
them friends of the pod.
TechStuff News
Hosts And Creators
Oz Woloshyn
Karah Preiss
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Joe Rogan Experience
The official podcast of comedian Joe Rogan.