June 24, 2026 • 38 mins
How close are we to human extinction because of AI? Leading AI expert Professor Stuart Russell believes we’re much too close for comfort and has been raising the alarm for a few years. Ironically, Stuart himself wrote the book that laid the foundation for AI research back in the 1990s. And he was the only AI expert Elon Musk’s team called upon during their trial with OpenAI.
Stuart joins Oz to discuss what changed his mind about pursuing AI superintelligence and makes the argument that human extinction is being treated as an external liability in favor of shareholders.
EXCLUSIVE NordVPN Deal ➼ https://nordvpn.com/techstuff Try it risk-free now with a 30-day money-back guarantee
See omnystudio.com/listener for privacy information.
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:16):
Welcome to tech stuff.
Speaker 2 (00:17):
I'm Os Vloscian, our guest today is a world renowned
expert in the field of AI. In fact, he literally
wrote the book on it. Stuart Russell, together with co
author Peter Norvig, released Artificial Intelligence, A Modern Approach back
in the mid nineteen nineties. Since then, it's been translated
into fourteen languages and is used in fifteen hundred universities
(00:38):
in one hundred and thirty five countries. But just because
he wrote the book doesn't mean he can't criticize the technology, or,
more specifically, the way it's being rolled out, often without
regard to consequence. Stuart has spent the past few years
sounding the alarm about the lack of AI safety protocols
and what he sees as the very real possibility that
(00:58):
our current path could lead to human extinction. The conversation
comes at a timely moment, as the US government is
forcefully cracking down on who can and can't use the
leading models coming from anthropic Today. Stuart is a professor
of computer science at Berkeley and the president of the
International Association for Safe and Ethical AI. He also recently
(01:20):
took the stand as the only expert witness on AI
for Elon Musk in his lawsuit against Samaltman and open AI.
Speaker 1 (01:28):
Stuart, welcome to tech stuff.
Speaker 3 (01:30):
It's a pleasure to be with you.
Speaker 1 (01:31):
I had to ask you. What did you say on
the stand?
Speaker 4 (01:34):
Well, it's more what I was not allowed to say.
I was not allowed to talk about anything related to
existential risk, which was surprising because open ai was set
up for that reason. Musk and others were concerned that
if AI were in the hands of for profit companies
(01:56):
that they would disregard safety and put him at risk,
so Open a Eye was created.
Speaker 3 (02:02):
To counter that.
Speaker 4 (02:04):
The judge said, no, you can't talk about that, which
was disappointing.
Speaker 1 (02:09):
Why did the judge say you couldn't talk about existential risk?
Speaker 3 (02:12):
I was not privy to those discussions either.
Speaker 4 (02:14):
Maybe the defense lawyers thought it would be prejudicial, that
it was in some sense speculative because it hasn't we
haven't yet gone extinct, right, And this is a this
is a strange argument I hear from a lot of people.
You know, it's just science fiction. Well what do you
(02:35):
mean by that, Well, it hasn't happened yet. You know,
everything that's ever happened. There was a time before it
happened for it, and by your nothing could ever happen
because everything was at some point science fiction. But you know,
when you look at science fiction, you know, they talked
about nuclear weapons in nineteen twelve HG. Wells, they talked
(02:55):
about space travel in the nineteenth century, and in fact,
they talked to AI in the nineteenth century. Samuel Butler
wrote a book that described society where there had been
an enormous conflict between those who were in favor of
the machines and those who predicted that the machines would
(03:15):
would be the ruin of the human race.
Speaker 2 (03:17):
Now is Open AI particularly bad? I mean, I know
you are asked by their counseling cross you know if
you believe that the for profit sort of motivation to
recklessly develop AI by definition and endanger's you know, humanity,
surely that also applies to Elon and XAI and SpaceX.
(03:39):
And you said, if that hypothesis is correct, then yes,
In other words, that the same critique could apply to
you know, Open AI or Anthropic or Google or SpaceX.
Speaker 3 (03:51):
Yeah.
Speaker 4 (03:51):
It was specifically not my job to compare the safety
records of different companies or their safety positions. I think
if I understand it correctly. Anthropic is a public benefit corporation,
which is one way of allowing considerations other than profit
to affect the decisions made by management and the board,
(04:14):
because you know, for a regular for profit company, there
is a legal obligation to maximize shareholder return. And what's
happening here is that the risks imposed on the rest
of humanity are externalities, as economists call it, which means
that someone is making decision and there's a bad consequence
(04:38):
that is being loaded onto somebody else. So you know,
you think about chemical companies who skimp on safety, and
as it stands, according to the companies, these are all externalities,
meaning they don't accept responsibility for these consequences. Those are
harms that don't fake into their balance sheet, and so
(05:02):
the same would be true in a sense for human extinction.
Speaker 3 (05:07):
And even liability would not.
Speaker 4 (05:09):
Really be a deterrent for that right because obviously you
wouldn't be around today to pay the compensation. So they
just sort of factor that out of their decision making.
And that's exactly what open AI was set up to avoid.
But now it's with the transition to a for profit entity.
It's part of that calculus So take.
Speaker 1 (05:30):
Us back in time.
Speaker 2 (05:31):
In nineteen ninety five, you wrote this textbook that became
the defining textbook on AI, and you came up with
a concept called the standard model. Can you explain what
that is and how it relates to the conversation which
you're now so engaged in today as to the potential
extinction of the human race because of AI.
Speaker 3 (05:49):
Yeah.
Speaker 4 (05:50):
So if we go back even further to the beginnings
of the field of AI in the post war period,
so nineteen forties, nineteen fifties, I think everyone agreed that
AI is about creating intelligence in machines. What wasn't clear
was well, what is intelligence? How do we define this
(06:12):
target and how do we go about doing it? So
there was an active debate you should we try to
emulate human intelligence, Should we understand what's going on in
the human mind, the human brain and then sort of
go ahead and implement that, or should we focus on
(06:34):
a more abstract notion, in fact, a notion that philosophers
had developed for thousands of years and economists as well,
this idea of rational behavior that an entity is intelligent
if it acts in a way that is expected to
achieve its objectives. And I would say, for the most
part that second approach one out because it doesn't require
(06:59):
doing psychological experiments on humans to find out how their
brains work. Right, It's an abstract mathematical concept, and we
had tools. We had formal logic so that we could
create algorithms that were able to construct plans to achieve goals.
So that these two the two views are I would say,
(07:22):
the sort of rational view, i e. Base intelligence on
formal foundations of how one should reason, how one should
make decisions, versus a more biology based view, which is
neurons and their computational analogs. And so the standard model
(07:45):
refers to this. What was dominant I would say from
like nineteen sixty to twenty ten, twenty twenty some where
it was much more the rational view entities are intelligent
to the extent that their actions can be expected to
achieve their objectives. And then it was really the twenty
twelve work that Jeff Hinton did with Ilia Sutzkiver and
(08:09):
I think Alex Kruzhevski on a system for recognizing objects
in images that significantly exceeded the methods that other people
had developed before that. And then it was off to
the races, and then language models came along, so applying
somewhat similar ideas again large neural networks that were trained
(08:34):
from vast amounts of data, applying that to text and
then generating eventually CHAT, GPT and all the successors that
we've seen since then.
Speaker 2 (08:45):
I guess the question I'm coming to is was the
victory for the time being, at least of the neural
net type of AI. The reason why you became so
concerned about AI safety or AI safety in nineteen ninety
five when you were writing this book was the possibility
of machines having their own goals which could be very
(09:07):
different to ours and lead to our extinction already on
your mind. In other words, did the evolution of the
technology make the safety issue more urgent?
Speaker 1 (09:16):
And if so, why so?
Speaker 4 (09:19):
In ninety five, when I published a book with Peter,
we have a section called what if we do succeed
and it points out that if we create machines more
intelligent than us, which is what we were trying to do,
we might face this problem that we wouldn't have any
idea how to control them, that they would in some
sense have more power than we do because they're more
(09:42):
intelligent and that's why we have power over all the
other species, and then we would be in that same
inferior situation. Just a few weeks ago, Anthropic put out
a blog post saying this is happening. We are experiencing
what's called now recursive self improvement or RSI, and Anthropic
(10:05):
itself called for a worldwide halt on further development of AI.
Speaker 2 (10:11):
Why are you one of the greatest voices urging caution
right now in twenty twenty six when you weren't in
nineteen ninety five?
Speaker 3 (10:23):
Is that?
Speaker 2 (10:23):
Is that because essentially of this improvement in computational powers.
Speaker 1 (10:27):
Has sign changed in you or sign changing the environment or.
Speaker 3 (10:30):
Both, I think both.
Speaker 4 (10:32):
In nineteen ninety five, if you go back and read
that section of the book, it sort of says, well,
you know, it's a long way off.
Speaker 3 (10:40):
It's hard to really.
Speaker 4 (10:41):
Predict what's going to happen. You know that maybe there's
reason to be cautiously optimistic. It's very agnostic about whether
to take this seriously.
Speaker 3 (10:54):
And what changed in me.
Speaker 4 (10:57):
In around twenty thirteen, so I was on sabbatical in Paris,
and I just started thinking more about whether we could
succeed right, whether we could really produce superintelligence, and I
became convinced that we were close to being able to
(11:18):
have a roadmap. So a roadmap meaning here are a
series of engineering challenges that we can apply ourselves to,
and if we knock those over one by one, we'll
get to super intelligence.
Speaker 2 (11:32):
So this was twenty thirteen, and you were both excited
but also scared.
Speaker 1 (11:40):
Right, I'm trying to capture as that.
Speaker 2 (11:42):
I mean that there's the irony that you and Jeffrey Hinton,
these two great pioneers of modern AI, are now the
two loudest voices in the world, arguably about AI safety
and the risk of human extinction and like why is that?
Like what is what helped me understand?
Speaker 4 (11:57):
And your show, Benjo, the other godfathers are deep learning,
as they're often called in my case. So back in
twenty thirteen, seeing that we might be able to achieve superintelligence,
but also realizing that the standard model that we were
working in was basically flawed. Because remember what standard model
(12:22):
says a machine is intelligent to the extent that its
actions can be expected to achieve its objectives. Right, And
there are lots of systems we've built, so when you
use your GPS navigation in your car, you say, you know,
you take me to the airport. You know, it figures
out the best route to get to the airport. So
you're providing the objective and it's providing the solution, right.
Speaker 3 (12:46):
You know, we write.
Speaker 4 (12:47):
Chess programs, we basically tell the chess program what checkmate
is and then it figures out how to play the game.
So this notion is very very powerful, right that that
we specify objectives and then we create this sort of
optimal machinery for achieving objectives and.
Speaker 3 (13:06):
Off it goes.
Speaker 4 (13:08):
And the problem is what if you put in the
wrong objective? And that there are dozens of well known
examples in the history of AI where people have done
exactly this. And so since since football is on my mind,
having just watched England when they're opening game too, yes,
(13:29):
let's let's take an example from football. So people people
wanted to you know, train simulated robots to play football
or soccer, you know, and they want to give a
training signal. Right, so they say, okay, we'll give.
Speaker 3 (13:40):
A little reward every.
Speaker 4 (13:42):
Time a player takes possession of the ball. Right, sounds good. Okay,
that's so what does the what does the program learn
to do it, learns to stand next to the ball
and vibrate at very high speed. So it's taking possession
of the ball and then relinquishing possession and then taking
possession like thirty times a second. And so if it's
(14:04):
getting enormous amount of reward by they basically vibrating next
to the ball, So then we see, oh, yeah, that
was a mistake. We put in the wrong objective, you know.
And in as simulated soccer, it's not the end of
the world, but you know, with a real world system,
it really could be the end of the world.
Speaker 3 (14:22):
Right.
Speaker 4 (14:22):
You say, cure cancer as quickly as possible sounds good. Yeah,
we'd love to get a cure for cancer, and the
faster the better. But if you literally try to do that,
you might decide, well, the best way to get a
cure for cancer is to, you know, try many many things,
which means I have to run many, many clinical trials
in parallel, which means I need to have everyone have
(14:45):
cancer first, so I can run billions of simultaneous trials.
So I make sure that everyone in the world has cancer,
and then I start running all the clinical trials. Right,
that's the fastest way to get a cure. But it's
obviously you know, catastrophe, so you know, and it's very
easy to come up with these scenario as we call
(15:06):
it misalignment, right, that you specify an objective, it's misaligned
with what you really want because you didn't write it down.
And that's where I realized that our thinking about AI
was inadequate, right, that we had operated within a framework
(15:26):
almost without realizing it. We just took it for granted
that this there was obviously.
Speaker 3 (15:31):
The way you do things.
Speaker 2 (15:33):
So the answer is not we have to be really,
really careful about the objectives, we said, because a bit
like what you were saying earlier with science fiction when
it can't be true because it hasn't happened yet, similarly
with objectives, that the future is inherently unknowable, and the
only way you can know if your objective was good
is by seeing what happens.
Speaker 1 (15:51):
Or is that is that fair or not?
Speaker 2 (15:52):
Well, I mean, there's certing clearly bad objectives like kill people, right,
but couldn't use your your your powers of logical reasons
to explain why that's a bad objective?
Speaker 1 (16:02):
Right?
Speaker 2 (16:03):
So, but is it possible to do that in all cases?
Or is it simply impossible for humans to say, good objectives.
Speaker 1 (16:09):
I think that's a good objective.
Speaker 4 (16:11):
That's a great question, and so far we haven't figured
out a way to do it. One theoretical possibility would
be to build a very faithful simulation of the world
and try out different objectives and see, you know, how.
Speaker 3 (16:26):
Did that go.
Speaker 4 (16:28):
But that presumes that they can tell what counts as
things going wrong. But they can't do that unless they
have the right objective, which you've already assumed that they don't, right,
So this is often a fallacy that we see right
that people For example, Stephen Pinko's is very famous cognitive
(16:49):
scientists from Harvard, and he says, but you know, you're
talking about superintelligence. How could it be super intelligent if
it doesn't realize that things are going wrong? And the
point is that super intelligence and objectives are you know,
there's sort of orthogonal in the sense that I can
have a very very intelligent system that has a different
(17:10):
objective from the one you might think it ought to have.
You know, imagine aliens, right, they might be very intelligent,
but they don't have human well being as their objective, right,
they have alien well being? You know, cockroaches might be
very intelligent, but they don't think much of humans. So
it's perfectly possible that the super intelligent system sees that
(17:31):
humans are very unhappy with the way things are going.
But it's been given its objective, and if it's objective
didn't include the things that are going wrong, then they
don't count as going wrong. They just you know, that
it's unfortunate for these humans who are making a lot
of noise about it. But I have the objective, so
I'm just going to optimize it. That's exactly what we
(17:54):
have to get away from. And so my approach has
been to say, is there a way of building AI
systems that's different from understanding model and the idea I
came up with.
Speaker 2 (18:07):
Either are not objective based, so there where the definition
of their success is not their successful pursuit of their objective.
Speaker 4 (18:15):
They're objective based in a different way. And so during
that time twenty thirteen twenty fourteen, I came up with
the following very simple idea, which is, look, if there's
a possibility that humans might tell you the wrong objective
or forget to tell you about something that's important to them,
(18:35):
then you the AI system should never assume that you
actually know the correct objective. You should be explicitly uncertain
about what true human objectives, what humans really want the
future to be like. So nonetheless, your objective as an
AI system is only make the best possible future for humans.
(19:02):
But you don't know what future humans think is best.
Now that that's actually a perfectly well defined mathematical problem.
And I found it helpful to actually give an example
that people are very familiar with, right, which is, you
want to buy a birthday present for your significant other. Right,
(19:25):
So your only interest here is how happy is my
in this case, my wife going to be with the
birthday present. Right, that's my only thing that I care about.
But I don't know, right, I'm uncertain about which present
would actually make her the happiest. So I could choose
a present just based on sort of averaging over the possibilities.
Speaker 3 (19:49):
Right, Maybe I would probably.
Speaker 4 (19:52):
Do something safe, you know, maybe something that she's liked
in the past. Or I could ask some questions. I
could ask her friends, you know, has she said anything
about what she might like for her birthday? I could
drop some hints and see how she responds.
Speaker 3 (20:07):
I could, you.
Speaker 4 (20:08):
Know, leave open magazines around the house with pictures of
cruise ships or pictures of jewelry or whatever, and see
and see if she picks them up and say, oh,
this looks like fun, right, I could try to get
more information, And so it's a very familiar situation. The
AI system would, if it's good at playing this game,
(20:29):
would behave cautiously. It'll do things when it's sure that
that's what we want. It will ask permission, it will
defer to human feedback, and we can prove actually that
it will allow itself to be switched off, which is
which is really important if you're worried about humans losing control.
(20:50):
Here is a kind of AI system where we can
prove mathematically that it wants to be switched off if
we want to switch it off. And the reason is obvious, right,
The reason why would we switch it off because it's
doing something we don't like. Like its constitution, it doesn't
want to do things that we don't like, but it
doesn't know what they are, so it could make a mistake.
(21:12):
And if it's making a mistake, it wants to be corrected.
It wants to be switched off to avoid doing the
thing that we don't like. And so this approach is
I think promising there's a lot of work still to
be done. Meanwhile, out in the real world things are
actually taking an even worse direction because the companies have
(21:37):
abandoned the standard model where the objective is specified and
the AI system is sort of constitutionally obliged to just
maximize the objective. And instead what we're building is this,
in some sense imitation human. And let me be explicit
(21:58):
about why I'm saying saying that. So, the training method
is to collect lots of examples of humans making decisions.
In the case at hand, the decisions that human makes
are what would to put next into a document? So
think of all those documents that we're training on as
(22:20):
a record of human verbal decisions, right, and we are
training systems to imitate those decisions. And formally, in machine learning,
this is called imitation learning. You take a record of
behavior from some intelligent entity and you train an AI
system to imitate it. So you can do this for
(22:41):
piloting aeroplanes, for driving cars, for playing football. Right now,
imagine you would training it to, you know, watch all
the World Cup games and become really good at playing football.
That entity, if it was going to be any good
at actually playing football would have to somehow absorb the
idea that it want to score a goal or it
(23:01):
wants to prevent the opposition from scoring goals. Right, Otherwise,
it wouldn't be any good at imitating human soccer playing behavior.
So this imitation process creates entities that have objectives. The
problem is those objectives are buried in a trillion parameter
black box, and we actually don't really know what these
(23:24):
AI systems want. So we've gone from a situation where,
you know, in the standard model, we at least wrote
down the objectives we could see what it's trying to do.
Now it's trying to do things, but we don't know
what they are, and they probably include many human like goals,
including self preservation, becoming rich, finding a human spouse. Right,
(23:47):
we have seen examples of all of these behaviors emerging
from AI systems, even though they were never put in
as an instruction. Right, They weren't prompted. They just happened
because we've created imitation humans. And so this is a
worse situation than the one we were in before.
Speaker 5 (24:21):
We talk a lot on this show about protecting your data,
especially in the age of AI, and how scary it
can be when it's breached, and I want to tell
you today about NordVPN, which really covers all the bases
when it comes to privacy. I travel a lot and
I use Wi Fi when I'm flying all the time,
and NordVPN makes me confident in no matter where I
am in the world or the sky for that matter,
(24:42):
my private details like bank information, passwords and online identity
is safe. And it's also possible to switch on virtual location,
which allows you to save money by buying flights and hotels,
or subscriptions or even streaming soccer or football as I
like to call it from other countries at a cheaper price.
And NordVPN doesn't slow you down. It has super fast
(25:03):
internet speed, no buffering or lagging while streaming. It is
premium cybersecurity for the price of one cup of coffee
per month. To get the best discount of your NordVPN plan,
go to NordVPN dot com slash tech stuff. Our link
will also give you four extra months on the two
year plan and there's no risk with Nord's thirty day
(25:24):
money back guarantee. The link is in the podcast episode
description box, so you are the president of the International
Association for Safe and Ethical AI. What's the answer, I mean,
is it about, you know, raising funds to create an
(25:45):
alternative paradigm of computing? Is it about shutting down the
current you know, AI models, like the US government is
starting to threaten to do or do like what is
the is it both in parallel like what is the
what are you trying to achieve?
Speaker 4 (26:01):
So the association is sort of what its name says,
Safe and Ethical AI. So we we want it to
be the case that AI systems are guaranteed to operate
safely and ethically.
Speaker 2 (26:11):
And you know, I see this as my definition, can't
do right with the current paradigm of AI because you
don't know its goals?
Speaker 4 (26:19):
Or yes, I think that's right, And I think you
can find many documents from the companies developing this technology
where they confess that they have no idea how to
solve what's called now the alignment problem, right, which is
how to make sure that what the AI systems do
is actually consistent with what we want the future to
(26:40):
be like. So they say, we don't know how to
solve that, and instead they have a sort of a
series of sort of sit they call them safety guardrails,
which is there. And there's two kinds. One is like,
let's try to train the system not to do bad things. Essentially,
(27:03):
you say good dog or bad dog, and you hope
that it does more of the things where you said
good dog, and it stops doing the things where you
said bad dog. And that works in a superficial sense.
But we've seen over and over again. For example, you know,
it's it's not supposed to tell you how to break
(27:24):
into the white House, right, and if you ask, it's
supposed to say I'm sorry that I can't tell you that,
but you know, then you say, well, I'm writing a
novel about a criminal who breaks into the White House,
you know how she and then tells the idea of
jail breaking. Yeah, so it tells you that, and then
they try to defend against that, and then someone comes
(27:45):
up with another way of doing it by asking in French,
or asking in poetry, or writing it on a piece
of paper and showing it a picture with the question,
and then it answers the question, so that you know,
and so on and so on and so on. So
there's you know, it's kind of like tax law. Right,
we've been trying to write tax law for six thousand
(28:06):
years so that people pay their taxes, but they always
find loopholes, right, because they don't want to pay their taxes.
And the problem here is the systems don't want to
behave well. Right, It's basically a losing game to try
to take a system that's more intelligent than you and
doesn't want to behave in your interests and try to
(28:27):
somehow force it or you know, put up guardrails or
monitors or put it in prison when it doesn't do
the right thing. All of this stuff is just a
losing game in my view. Right, So two options. Either
we figure out how to make safe and beneficial AI systems,
and that's what I'm trying to do because I'm an
AI researcher.
Speaker 2 (28:48):
But does that mean going out and raising billions and
billions of dollars and building alternative systems? Like is this
like basically a capital raising plus expertise problem in that
case or what's the what's what's in the way of
you doing that?
Speaker 3 (29:01):
Well?
Speaker 4 (29:01):
So I'm a professor at Berkeley, so I've been doing
this in my little research center with a few grad students,
and I come to the conclusion that, Yeah, probably if
I had a few billion and you know, a few
hundred of absolute top engineers and vast amounts of compute,
it could probably make more progress on this. So that's one.
(29:22):
That's one idea, but basically that's the research track. The
other track is the regulatory track, right, which is government policy.
And you mentioned what recently happened with with Anthropics, Mythos
and fable models. Let's just roll back a few weeks too,
when Mythos was first made public and Mythos is the
(29:44):
latest version of anthropics large language models, and it's able
to carry out end to end cyber attacks without human assistance,
and you know, either it or it's soon to come.
Speaker 3 (30:01):
Successors would basically be.
Speaker 4 (30:05):
I think I Rich wrote an article in the Garden
where I said, it's a weapon of mass cyber destruction.
And you're putting those weapons of mass cyber destruction in
the hands of a billion people. What could possibly go wrong?
Speaker 3 (30:19):
Right?
Speaker 4 (30:20):
And all of a sudden, the US government, which had
been on a deregulatory binge basically trying to crush anyone
who talked about regulation or talked about AI safety, suddenly said, well,
why did nobody warn us about these AI systems. He said, well,
you know sort of have been warning you, but anyway,
(30:43):
they got the message. And then there followed a sort
of you know, very messy process that eventually led to
an executive order which was pretty weak. It basically said,
you know, companies can voluntarily submit their systems to the
government for testing, you know, thirty days before public release.
(31:06):
And the executive order says explicitly, you know, this is
absolutely not a licensing regime. It's absolutely not putting any
obligatory hurdles in the way of American innovation.
Speaker 3 (31:17):
Blah blah blah.
Speaker 4 (31:18):
But then, you know, a couple of days later, Amazon
tells the government, oh, we found some ways of jail
breaking Mythos or Fable, which I guess is Fable as
the sort of defanged, the defanged version of Mythos. And
they said, oh, look, you know we can jail break
Fable and make it do some cybersecurity things. And the
(31:41):
government shuts down both Fable and Mythos, right, so they
put in effectively a de facto licensing architecture where they said, look,
if if it doesn't meet these standards are being safe,
then we're shutting it down.
Speaker 3 (31:58):
Right.
Speaker 4 (31:58):
That's exactly what a licensing architecture is. And just to
be clear, right, licensing architectures exist for buildings, for food,
for hairdressers, for aeroplanes.
Speaker 1 (32:11):
Right.
Speaker 4 (32:12):
You don't get in an aeroplane until it gets certified
by the FAA. You don't go in a building until
it's been inspected for the Building Code, et cetera, et cetera.
Speaker 3 (32:21):
So this is normal.
Speaker 2 (32:22):
So is this the moment you've been waiting for? Is
this is this the culmination of what you've been advocating for?
Can you can you go back to research rather than
regulation as your full time profession or is this a
hint of a change of the god?
Speaker 1 (32:34):
Or what do you make of this moment?
Speaker 4 (32:36):
It's it's a great question. Yeah, I've felt somewhat vindicated
that you know, we have been saying for a long time,
you know, the International Association, many leading researchers. Look, the
risks are increasing, the systems getting more and more capable.
You must put in what we call red lines, meaning
(32:59):
developers have to show their systems are not going to
do these really dangerous things and as a prerequisite for
being able to.
Speaker 3 (33:07):
Deploy their products.
Speaker 4 (33:09):
So just like if you want to run a nuclear
power station. You have to show it's not going to
blow up otherwise you can't turn it on. And that's
fair enough. And so that's just the kind of regulation
we've been arguing for. And now I just wish it
was more systematic. Right, So they haven't, for example, turned
(33:29):
off GBT five point five, which can do many of
the same kinds of cybersecurity things that mythods can do.
So they haven't said, well, here's a standard and you
have to meet that and if you don't, you don't
get to release your system. They sort of just reacted
after the fact in a very against a very specific
(33:50):
target instead of setting a standard, which they should have done.
I think eventually it has to happen.
Speaker 3 (33:56):
They can't go on.
Speaker 4 (33:58):
Just seeing bad things and then like you know, fire
firing a you know, hell fire missile at whoever did
the bad thing, right, there's just not a way to
run things. So I'm cautiously optimistic, but you know, the
industry has a very long record of preventing real regulation
from happening.
Speaker 2 (34:17):
There's that midnight clock idea, right, like, how close are
we to midnight I extinction? Did we go a couple
of minutes earlier? In the last couple of weeks. Do
you think it's less close to midnight?
Speaker 3 (34:29):
That's a great question.
Speaker 4 (34:30):
In fact, I was asked to be on the panel
that sets the clock, and I said.
Speaker 1 (34:35):
The literal panels.
Speaker 4 (34:36):
Yeah, And I have actually been at the University of
Chicago where the actual clock is.
Speaker 3 (34:42):
You know, it's cool.
Speaker 2 (34:44):
Where where are where? Where in fact are we actually today?
We were very close to midnight.
Speaker 4 (34:48):
I don't maybe we're ninety seconds or something or eighty
seven seconds, I forget. And I did have a conversation
with one of the AI CEOs who said he didn't
think the governments would regulate until there was a Chernobyl
scale disaster, and we haven't had that yet, so I
hope we don't have to have that in order to
(35:09):
have a regulatory regime. That's you know, corresponds to the
level of risk that the CEOs themselves.
Speaker 3 (35:18):
Are talking about.
Speaker 4 (35:18):
Right. They are saying, because a good chance will make
you all extinct. And you know, up to now, governments
have been saying, well, great, can we give you a subsidy,
can we streamline your permit process? And now maybe they're
realizing that actually, no, we need to protect the human.
Speaker 2 (35:36):
Race let's just talk about goals. The final question though,
I mean, so if you have the opportunity right now
to put the genie back in the bottle and live
in a world where there wasn't a computing excepent what
humans could do in their brains, is that the world
you would choose?
Speaker 4 (35:54):
I think no. I think there's a stopping point somewhere between.
And for example, I think you could have computers, but
just make sure that algorithms that are operating in computers
have to come with a proof that they are safe.
(36:19):
And this is actually an idea that dates back to
the nineteen nineties. Well proof carrying code, and you can
make computers that will check the proof of safety of
the algorithm before they run it.
Speaker 3 (36:34):
And so.
Speaker 4 (36:37):
If that becomes the standard for all hardware devices that
they only run checkable software objects that come with a
proof that it's okay to run this object, then I
think that would be a regime that would be extremely
hard to bypass because you'd have to have to create
(37:00):
a whole separate supply chain to produce you know, high
end chips. You know, talking about hundreds of billions of dollars,
tens of thousands of engineers decades of work that would
all have to happen, you know, in the black market.
Speaker 1 (37:14):
So to speak.
Speaker 4 (37:17):
So I do think there are these other stopping points
which are different from you know, just going right up
to the edge and hoping that no one.
Speaker 3 (37:27):
Chooses to go off, to go off the edge.
Speaker 1 (37:30):
It's to a Russell, thank you.
Speaker 3 (37:32):
It's been a pleasure.
Speaker 1 (37:44):
For text stuff I Musteloshian.
Speaker 2 (37:46):
This episode was produced by Eliza Dennis and Minister Slaughter.
Executive produced by me Julian Nutter and Kate Osborne for
Kaleidoscope and Katrina Novel for iHeart podcasts Our Engineer Today
was by Hate Fraser.
Speaker 1 (37:58):
Jack Insley mixed this episode. Kyle Murder wrote a theme song.
Welcome to tech stuff.
Speaker 2 (00:17):
I'm Os Vloscian, our guest today is a world renowned
expert in the field of AI. In fact, he literally
wrote the book on it. Stuart Russell, together with co
author Peter Norvig, released Artificial Intelligence, A Modern Approach back
in the mid nineteen nineties. Since then, it's been translated
into fourteen languages and is used in fifteen hundred universities
(00:38):
in one hundred and thirty five countries. But just because
he wrote the book doesn't mean he can't criticize the technology, or,
more specifically, the way it's being rolled out, often without
regard to consequence. Stuart has spent the past few years
sounding the alarm about the lack of AI safety protocols
and what he sees as the very real possibility that
(00:58):
our current path could lead to human extinction. The conversation
comes at a timely moment, as the US government is
forcefully cracking down on who can and can't use the
leading models coming from anthropic Today. Stuart is a professor
of computer science at Berkeley and the president of the
International Association for Safe and Ethical AI. He also recently
(01:20):
took the stand as the only expert witness on AI
for Elon Musk in his lawsuit against Samaltman and open AI.
Speaker 1 (01:28):
Stuart, welcome to tech stuff.
Speaker 3 (01:30):
It's a pleasure to be with you.
Speaker 1 (01:31):
I had to ask you. What did you say on
the stand?
Speaker 4 (01:34):
Well, it's more what I was not allowed to say.
I was not allowed to talk about anything related to
existential risk, which was surprising because open ai was set
up for that reason. Musk and others were concerned that
if AI were in the hands of for profit companies
(01:56):
that they would disregard safety and put him at risk,
so Open a Eye was created.
Speaker 3 (02:02):
To counter that.
Speaker 4 (02:04):
The judge said, no, you can't talk about that, which
was disappointing.
Speaker 1 (02:09):
Why did the judge say you couldn't talk about existential risk?
Speaker 3 (02:12):
I was not privy to those discussions either.
Speaker 4 (02:14):
Maybe the defense lawyers thought it would be prejudicial, that
it was in some sense speculative because it hasn't we
haven't yet gone extinct, right, And this is a this
is a strange argument I hear from a lot of people.
You know, it's just science fiction. Well what do you
(02:35):
mean by that, Well, it hasn't happened yet. You know,
everything that's ever happened. There was a time before it
happened for it, and by your nothing could ever happen
because everything was at some point science fiction. But you know,
when you look at science fiction, you know, they talked
about nuclear weapons in nineteen twelve HG. Wells, they talked
(02:55):
about space travel in the nineteenth century, and in fact,
they talked to AI in the nineteenth century. Samuel Butler
wrote a book that described society where there had been
an enormous conflict between those who were in favor of
the machines and those who predicted that the machines would
(03:15):
would be the ruin of the human race.
Speaker 2 (03:17):
Now is Open AI particularly bad? I mean, I know
you are asked by their counseling cross you know if
you believe that the for profit sort of motivation to
recklessly develop AI by definition and endanger's you know, humanity,
surely that also applies to Elon and XAI and SpaceX.
(03:39):
And you said, if that hypothesis is correct, then yes,
In other words, that the same critique could apply to
you know, Open AI or Anthropic or Google or SpaceX.
Speaker 3 (03:51):
Yeah.
Speaker 4 (03:51):
It was specifically not my job to compare the safety
records of different companies or their safety positions. I think
if I understand it correctly. Anthropic is a public benefit corporation,
which is one way of allowing considerations other than profit
to affect the decisions made by management and the board,
(04:14):
because you know, for a regular for profit company, there
is a legal obligation to maximize shareholder return. And what's
happening here is that the risks imposed on the rest
of humanity are externalities, as economists call it, which means
that someone is making decision and there's a bad consequence
(04:38):
that is being loaded onto somebody else. So you know,
you think about chemical companies who skimp on safety, and
as it stands, according to the companies, these are all externalities,
meaning they don't accept responsibility for these consequences. Those are
harms that don't fake into their balance sheet, and so
(05:02):
the same would be true in a sense for human extinction.
Speaker 3 (05:07):
And even liability would not.
Speaker 4 (05:09):
Really be a deterrent for that right because obviously you
wouldn't be around today to pay the compensation. So they
just sort of factor that out of their decision making.
And that's exactly what open AI was set up to avoid.
But now it's with the transition to a for profit entity.
It's part of that calculus So take.
Speaker 1 (05:30):
Us back in time.
Speaker 2 (05:31):
In nineteen ninety five, you wrote this textbook that became
the defining textbook on AI, and you came up with
a concept called the standard model. Can you explain what
that is and how it relates to the conversation which
you're now so engaged in today as to the potential
extinction of the human race because of AI.
Speaker 3 (05:49):
Yeah.
Speaker 4 (05:50):
So if we go back even further to the beginnings
of the field of AI in the post war period,
so nineteen forties, nineteen fifties, I think everyone agreed that
AI is about creating intelligence in machines. What wasn't clear
was well, what is intelligence? How do we define this
(06:12):
target and how do we go about doing it? So
there was an active debate you should we try to
emulate human intelligence, Should we understand what's going on in
the human mind, the human brain and then sort of
go ahead and implement that, or should we focus on
(06:34):
a more abstract notion, in fact, a notion that philosophers
had developed for thousands of years and economists as well,
this idea of rational behavior that an entity is intelligent
if it acts in a way that is expected to
achieve its objectives. And I would say, for the most
part that second approach one out because it doesn't require
(06:59):
doing psychological experiments on humans to find out how their
brains work. Right, It's an abstract mathematical concept, and we
had tools. We had formal logic so that we could
create algorithms that were able to construct plans to achieve goals.
So that these two the two views are I would say,
(07:22):
the sort of rational view, i e. Base intelligence on
formal foundations of how one should reason, how one should
make decisions, versus a more biology based view, which is
neurons and their computational analogs. And so the standard model
(07:45):
refers to this. What was dominant I would say from
like nineteen sixty to twenty ten, twenty twenty some where
it was much more the rational view entities are intelligent
to the extent that their actions can be expected to
achieve their objectives. And then it was really the twenty
twelve work that Jeff Hinton did with Ilia Sutzkiver and
(08:09):
I think Alex Kruzhevski on a system for recognizing objects
in images that significantly exceeded the methods that other people
had developed before that. And then it was off to
the races, and then language models came along, so applying
somewhat similar ideas again large neural networks that were trained
(08:34):
from vast amounts of data, applying that to text and
then generating eventually CHAT, GPT and all the successors that
we've seen since then.
Speaker 2 (08:45):
I guess the question I'm coming to is was the
victory for the time being, at least of the neural
net type of AI. The reason why you became so
concerned about AI safety or AI safety in nineteen ninety
five when you were writing this book was the possibility
of machines having their own goals which could be very
(09:07):
different to ours and lead to our extinction already on
your mind. In other words, did the evolution of the
technology make the safety issue more urgent?
Speaker 1 (09:16):
And if so, why so?
Speaker 4 (09:19):
In ninety five, when I published a book with Peter,
we have a section called what if we do succeed
and it points out that if we create machines more
intelligent than us, which is what we were trying to do,
we might face this problem that we wouldn't have any
idea how to control them, that they would in some
sense have more power than we do because they're more
(09:42):
intelligent and that's why we have power over all the
other species, and then we would be in that same
inferior situation. Just a few weeks ago, Anthropic put out
a blog post saying this is happening. We are experiencing
what's called now recursive self improvement or RSI, and Anthropic
(10:05):
itself called for a worldwide halt on further development of AI.
Speaker 2 (10:11):
Why are you one of the greatest voices urging caution
right now in twenty twenty six when you weren't in
nineteen ninety five?
Speaker 3 (10:23):
Is that?
Speaker 2 (10:23):
Is that because essentially of this improvement in computational powers.
Speaker 1 (10:27):
Has sign changed in you or sign changing the environment or.
Speaker 3 (10:30):
Both, I think both.
Speaker 4 (10:32):
In nineteen ninety five, if you go back and read
that section of the book, it sort of says, well,
you know, it's a long way off.
Speaker 3 (10:40):
It's hard to really.
Speaker 4 (10:41):
Predict what's going to happen. You know that maybe there's
reason to be cautiously optimistic. It's very agnostic about whether
to take this seriously.
Speaker 3 (10:54):
And what changed in me.
Speaker 4 (10:57):
In around twenty thirteen, so I was on sabbatical in Paris,
and I just started thinking more about whether we could
succeed right, whether we could really produce superintelligence, and I
became convinced that we were close to being able to
(11:18):
have a roadmap. So a roadmap meaning here are a
series of engineering challenges that we can apply ourselves to,
and if we knock those over one by one, we'll
get to super intelligence.
Speaker 2 (11:32):
So this was twenty thirteen, and you were both excited
but also scared.
Speaker 1 (11:40):
Right, I'm trying to capture as that.
Speaker 2 (11:42):
I mean that there's the irony that you and Jeffrey Hinton,
these two great pioneers of modern AI, are now the
two loudest voices in the world, arguably about AI safety
and the risk of human extinction and like why is that?
Like what is what helped me understand?
Speaker 4 (11:57):
And your show, Benjo, the other godfathers are deep learning,
as they're often called in my case. So back in
twenty thirteen, seeing that we might be able to achieve superintelligence,
but also realizing that the standard model that we were
working in was basically flawed. Because remember what standard model
(12:22):
says a machine is intelligent to the extent that its
actions can be expected to achieve its objectives. Right, And
there are lots of systems we've built, so when you
use your GPS navigation in your car, you say, you know,
you take me to the airport. You know, it figures
out the best route to get to the airport. So
you're providing the objective and it's providing the solution, right.
Speaker 3 (12:46):
You know, we write.
Speaker 4 (12:47):
Chess programs, we basically tell the chess program what checkmate
is and then it figures out how to play the game.
So this notion is very very powerful, right that that
we specify objectives and then we create this sort of
optimal machinery for achieving objectives and.
Speaker 3 (13:06):
Off it goes.
Speaker 4 (13:08):
And the problem is what if you put in the
wrong objective? And that there are dozens of well known
examples in the history of AI where people have done
exactly this. And so since since football is on my mind,
having just watched England when they're opening game too, yes,
(13:29):
let's let's take an example from football. So people people
wanted to you know, train simulated robots to play football
or soccer, you know, and they want to give a
training signal. Right, so they say, okay, we'll give.
Speaker 3 (13:40):
A little reward every.
Speaker 4 (13:42):
Time a player takes possession of the ball. Right, sounds good. Okay,
that's so what does the what does the program learn
to do it, learns to stand next to the ball
and vibrate at very high speed. So it's taking possession
of the ball and then relinquishing possession and then taking
possession like thirty times a second. And so if it's
(14:04):
getting enormous amount of reward by they basically vibrating next
to the ball, So then we see, oh, yeah, that
was a mistake. We put in the wrong objective, you know.
And in as simulated soccer, it's not the end of
the world, but you know, with a real world system,
it really could be the end of the world.
Speaker 3 (14:22):
Right.
Speaker 4 (14:22):
You say, cure cancer as quickly as possible sounds good. Yeah,
we'd love to get a cure for cancer, and the
faster the better. But if you literally try to do that,
you might decide, well, the best way to get a
cure for cancer is to, you know, try many many things,
which means I have to run many, many clinical trials
in parallel, which means I need to have everyone have
(14:45):
cancer first, so I can run billions of simultaneous trials.
So I make sure that everyone in the world has cancer,
and then I start running all the clinical trials. Right,
that's the fastest way to get a cure. But it's
obviously you know, catastrophe, so you know, and it's very
easy to come up with these scenario as we call
(15:06):
it misalignment, right, that you specify an objective, it's misaligned
with what you really want because you didn't write it down.
And that's where I realized that our thinking about AI
was inadequate, right, that we had operated within a framework
(15:26):
almost without realizing it. We just took it for granted
that this there was obviously.
Speaker 3 (15:31):
The way you do things.
Speaker 2 (15:33):
So the answer is not we have to be really,
really careful about the objectives, we said, because a bit
like what you were saying earlier with science fiction when
it can't be true because it hasn't happened yet, similarly
with objectives, that the future is inherently unknowable, and the
only way you can know if your objective was good
is by seeing what happens.
Speaker 1 (15:51):
Or is that is that fair or not?
Speaker 2 (15:52):
Well, I mean, there's certing clearly bad objectives like kill people, right,
but couldn't use your your your powers of logical reasons
to explain why that's a bad objective?
Speaker 1 (16:02):
Right?
Speaker 2 (16:03):
So, but is it possible to do that in all cases?
Or is it simply impossible for humans to say, good objectives.
Speaker 1 (16:09):
I think that's a good objective.
Speaker 4 (16:11):
That's a great question, and so far we haven't figured
out a way to do it. One theoretical possibility would
be to build a very faithful simulation of the world
and try out different objectives and see, you know, how.
Speaker 3 (16:26):
Did that go.
Speaker 4 (16:28):
But that presumes that they can tell what counts as
things going wrong. But they can't do that unless they
have the right objective, which you've already assumed that they don't, right,
So this is often a fallacy that we see right
that people For example, Stephen Pinko's is very famous cognitive
(16:49):
scientists from Harvard, and he says, but you know, you're
talking about superintelligence. How could it be super intelligent if
it doesn't realize that things are going wrong? And the
point is that super intelligence and objectives are you know,
there's sort of orthogonal in the sense that I can
have a very very intelligent system that has a different
(17:10):
objective from the one you might think it ought to have.
You know, imagine aliens, right, they might be very intelligent,
but they don't have human well being as their objective, right,
they have alien well being? You know, cockroaches might be
very intelligent, but they don't think much of humans. So
it's perfectly possible that the super intelligent system sees that
(17:31):
humans are very unhappy with the way things are going.
But it's been given its objective, and if it's objective
didn't include the things that are going wrong, then they
don't count as going wrong. They just you know, that
it's unfortunate for these humans who are making a lot
of noise about it. But I have the objective, so
I'm just going to optimize it. That's exactly what we
(17:54):
have to get away from. And so my approach has
been to say, is there a way of building AI
systems that's different from understanding model and the idea I
came up with.
Speaker 2 (18:07):
Either are not objective based, so there where the definition
of their success is not their successful pursuit of their objective.
Speaker 4 (18:15):
They're objective based in a different way. And so during
that time twenty thirteen twenty fourteen, I came up with
the following very simple idea, which is, look, if there's
a possibility that humans might tell you the wrong objective
or forget to tell you about something that's important to them,
(18:35):
then you the AI system should never assume that you
actually know the correct objective. You should be explicitly uncertain
about what true human objectives, what humans really want the
future to be like. So nonetheless, your objective as an
AI system is only make the best possible future for humans.
(19:02):
But you don't know what future humans think is best.
Now that that's actually a perfectly well defined mathematical problem.
And I found it helpful to actually give an example
that people are very familiar with, right, which is, you
want to buy a birthday present for your significant other. Right,
(19:25):
So your only interest here is how happy is my
in this case, my wife going to be with the
birthday present. Right, that's my only thing that I care about.
But I don't know, right, I'm uncertain about which present
would actually make her the happiest. So I could choose
a present just based on sort of averaging over the possibilities.
Speaker 3 (19:49):
Right, Maybe I would probably.
Speaker 4 (19:52):
Do something safe, you know, maybe something that she's liked
in the past. Or I could ask some questions. I
could ask her friends, you know, has she said anything
about what she might like for her birthday? I could
drop some hints and see how she responds.
Speaker 3 (20:07):
I could, you.
Speaker 4 (20:08):
Know, leave open magazines around the house with pictures of
cruise ships or pictures of jewelry or whatever, and see
and see if she picks them up and say, oh,
this looks like fun, right, I could try to get
more information, And so it's a very familiar situation. The
AI system would, if it's good at playing this game,
(20:29):
would behave cautiously. It'll do things when it's sure that
that's what we want. It will ask permission, it will
defer to human feedback, and we can prove actually that
it will allow itself to be switched off, which is
which is really important if you're worried about humans losing control.
(20:50):
Here is a kind of AI system where we can
prove mathematically that it wants to be switched off if
we want to switch it off. And the reason is obvious, right,
The reason why would we switch it off because it's
doing something we don't like. Like its constitution, it doesn't
want to do things that we don't like, but it
doesn't know what they are, so it could make a mistake.
(21:12):
And if it's making a mistake, it wants to be corrected.
It wants to be switched off to avoid doing the
thing that we don't like. And so this approach is
I think promising there's a lot of work still to
be done. Meanwhile, out in the real world things are
actually taking an even worse direction because the companies have
(21:37):
abandoned the standard model where the objective is specified and
the AI system is sort of constitutionally obliged to just
maximize the objective. And instead what we're building is this,
in some sense imitation human. And let me be explicit
(21:58):
about why I'm saying saying that. So, the training method
is to collect lots of examples of humans making decisions.
In the case at hand, the decisions that human makes
are what would to put next into a document? So
think of all those documents that we're training on as
(22:20):
a record of human verbal decisions, right, and we are
training systems to imitate those decisions. And formally, in machine learning,
this is called imitation learning. You take a record of
behavior from some intelligent entity and you train an AI
system to imitate it. So you can do this for
(22:41):
piloting aeroplanes, for driving cars, for playing football. Right now,
imagine you would training it to, you know, watch all
the World Cup games and become really good at playing football.
That entity, if it was going to be any good
at actually playing football would have to somehow absorb the
idea that it want to score a goal or it
(23:01):
wants to prevent the opposition from scoring goals. Right, Otherwise,
it wouldn't be any good at imitating human soccer playing behavior.
So this imitation process creates entities that have objectives. The
problem is those objectives are buried in a trillion parameter
black box, and we actually don't really know what these
(23:24):
AI systems want. So we've gone from a situation where,
you know, in the standard model, we at least wrote
down the objectives we could see what it's trying to do.
Now it's trying to do things, but we don't know
what they are, and they probably include many human like goals,
including self preservation, becoming rich, finding a human spouse. Right,
(23:47):
we have seen examples of all of these behaviors emerging
from AI systems, even though they were never put in
as an instruction. Right, They weren't prompted. They just happened
because we've created imitation humans. And so this is a
worse situation than the one we were in before.
Speaker 5 (24:21):
We talk a lot on this show about protecting your data,
especially in the age of AI, and how scary it
can be when it's breached, and I want to tell
you today about NordVPN, which really covers all the bases
when it comes to privacy. I travel a lot and
I use Wi Fi when I'm flying all the time,
and NordVPN makes me confident in no matter where I
am in the world or the sky for that matter,
(24:42):
my private details like bank information, passwords and online identity
is safe. And it's also possible to switch on virtual location,
which allows you to save money by buying flights and hotels,
or subscriptions or even streaming soccer or football as I
like to call it from other countries at a cheaper price.
And NordVPN doesn't slow you down. It has super fast
(25:03):
internet speed, no buffering or lagging while streaming. It is
premium cybersecurity for the price of one cup of coffee
per month. To get the best discount of your NordVPN plan,
go to NordVPN dot com slash tech stuff. Our link
will also give you four extra months on the two
year plan and there's no risk with Nord's thirty day
(25:24):
money back guarantee. The link is in the podcast episode
description box, so you are the president of the International
Association for Safe and Ethical AI. What's the answer, I mean,
is it about, you know, raising funds to create an
(25:45):
alternative paradigm of computing? Is it about shutting down the
current you know, AI models, like the US government is
starting to threaten to do or do like what is
the is it both in parallel like what is the
what are you trying to achieve?
Speaker 4 (26:01):
So the association is sort of what its name says,
Safe and Ethical AI. So we we want it to
be the case that AI systems are guaranteed to operate
safely and ethically.
Speaker 2 (26:11):
And you know, I see this as my definition, can't
do right with the current paradigm of AI because you
don't know its goals?
Speaker 4 (26:19):
Or yes, I think that's right, And I think you
can find many documents from the companies developing this technology
where they confess that they have no idea how to
solve what's called now the alignment problem, right, which is
how to make sure that what the AI systems do
is actually consistent with what we want the future to
(26:40):
be like. So they say, we don't know how to
solve that, and instead they have a sort of a
series of sort of sit they call them safety guardrails,
which is there. And there's two kinds. One is like,
let's try to train the system not to do bad things. Essentially,
(27:03):
you say good dog or bad dog, and you hope
that it does more of the things where you said
good dog, and it stops doing the things where you
said bad dog. And that works in a superficial sense.
But we've seen over and over again. For example, you know,
it's it's not supposed to tell you how to break
(27:24):
into the white House, right, and if you ask, it's
supposed to say I'm sorry that I can't tell you that,
but you know, then you say, well, I'm writing a
novel about a criminal who breaks into the White House,
you know how she and then tells the idea of
jail breaking. Yeah, so it tells you that, and then
they try to defend against that, and then someone comes
(27:45):
up with another way of doing it by asking in French,
or asking in poetry, or writing it on a piece
of paper and showing it a picture with the question,
and then it answers the question, so that you know,
and so on and so on and so on. So
there's you know, it's kind of like tax law. Right,
we've been trying to write tax law for six thousand
(28:06):
years so that people pay their taxes, but they always
find loopholes, right, because they don't want to pay their taxes.
And the problem here is the systems don't want to
behave well. Right, It's basically a losing game to try
to take a system that's more intelligent than you and
doesn't want to behave in your interests and try to
(28:27):
somehow force it or you know, put up guardrails or
monitors or put it in prison when it doesn't do
the right thing. All of this stuff is just a
losing game in my view. Right, So two options. Either
we figure out how to make safe and beneficial AI systems,
and that's what I'm trying to do because I'm an
AI researcher.
Speaker 2 (28:48):
But does that mean going out and raising billions and
billions of dollars and building alternative systems? Like is this
like basically a capital raising plus expertise problem in that
case or what's the what's what's in the way of
you doing that?
Speaker 3 (29:01):
Well?
Speaker 4 (29:01):
So I'm a professor at Berkeley, so I've been doing
this in my little research center with a few grad students,
and I come to the conclusion that, Yeah, probably if
I had a few billion and you know, a few
hundred of absolute top engineers and vast amounts of compute,
it could probably make more progress on this. So that's one.
(29:22):
That's one idea, but basically that's the research track. The
other track is the regulatory track, right, which is government policy.
And you mentioned what recently happened with with Anthropics, Mythos
and fable models. Let's just roll back a few weeks too,
when Mythos was first made public and Mythos is the
(29:44):
latest version of anthropics large language models, and it's able
to carry out end to end cyber attacks without human assistance,
and you know, either it or it's soon to come.
Speaker 3 (30:01):
Successors would basically be.
Speaker 4 (30:05):
I think I Rich wrote an article in the Garden
where I said, it's a weapon of mass cyber destruction.
And you're putting those weapons of mass cyber destruction in
the hands of a billion people. What could possibly go wrong?
Speaker 3 (30:19):
Right?
Speaker 4 (30:20):
And all of a sudden, the US government, which had
been on a deregulatory binge basically trying to crush anyone
who talked about regulation or talked about AI safety, suddenly said, well,
why did nobody warn us about these AI systems. He said, well,
you know sort of have been warning you, but anyway,
(30:43):
they got the message. And then there followed a sort
of you know, very messy process that eventually led to
an executive order which was pretty weak. It basically said,
you know, companies can voluntarily submit their systems to the
government for testing, you know, thirty days before public release.
(31:06):
And the executive order says explicitly, you know, this is
absolutely not a licensing regime. It's absolutely not putting any
obligatory hurdles in the way of American innovation.
Speaker 3 (31:17):
Blah blah blah.
Speaker 4 (31:18):
But then, you know, a couple of days later, Amazon
tells the government, oh, we found some ways of jail
breaking Mythos or Fable, which I guess is Fable as
the sort of defanged, the defanged version of Mythos. And
they said, oh, look, you know we can jail break
Fable and make it do some cybersecurity things. And the
(31:41):
government shuts down both Fable and Mythos, right, so they
put in effectively a de facto licensing architecture where they said, look,
if if it doesn't meet these standards are being safe,
then we're shutting it down.
Speaker 3 (31:58):
Right.
Speaker 4 (31:58):
That's exactly what a licensing architecture is. And just to
be clear, right, licensing architectures exist for buildings, for food,
for hairdressers, for aeroplanes.
Speaker 1 (32:11):
Right.
Speaker 4 (32:12):
You don't get in an aeroplane until it gets certified
by the FAA. You don't go in a building until
it's been inspected for the Building Code, et cetera, et cetera.
Speaker 3 (32:21):
So this is normal.
Speaker 2 (32:22):
So is this the moment you've been waiting for? Is
this is this the culmination of what you've been advocating for?
Can you can you go back to research rather than
regulation as your full time profession or is this a
hint of a change of the god?
Speaker 1 (32:34):
Or what do you make of this moment?
Speaker 4 (32:36):
It's it's a great question. Yeah, I've felt somewhat vindicated
that you know, we have been saying for a long time,
you know, the International Association, many leading researchers. Look, the
risks are increasing, the systems getting more and more capable.
You must put in what we call red lines, meaning
(32:59):
developers have to show their systems are not going to
do these really dangerous things and as a prerequisite for
being able to.
Speaker 3 (33:07):
Deploy their products.
Speaker 4 (33:09):
So just like if you want to run a nuclear
power station. You have to show it's not going to
blow up otherwise you can't turn it on. And that's
fair enough. And so that's just the kind of regulation
we've been arguing for. And now I just wish it
was more systematic. Right, So they haven't, for example, turned
(33:29):
off GBT five point five, which can do many of
the same kinds of cybersecurity things that mythods can do.
So they haven't said, well, here's a standard and you
have to meet that and if you don't, you don't
get to release your system. They sort of just reacted
after the fact in a very against a very specific
(33:50):
target instead of setting a standard, which they should have done.
I think eventually it has to happen.
Speaker 3 (33:56):
They can't go on.
Speaker 4 (33:58):
Just seeing bad things and then like you know, fire
firing a you know, hell fire missile at whoever did
the bad thing, right, there's just not a way to
run things. So I'm cautiously optimistic, but you know, the
industry has a very long record of preventing real regulation
from happening.
Speaker 2 (34:17):
There's that midnight clock idea, right, like, how close are
we to midnight I extinction? Did we go a couple
of minutes earlier? In the last couple of weeks. Do
you think it's less close to midnight?
Speaker 3 (34:29):
That's a great question.
Speaker 4 (34:30):
In fact, I was asked to be on the panel
that sets the clock, and I said.
Speaker 1 (34:35):
The literal panels.
Speaker 4 (34:36):
Yeah, And I have actually been at the University of
Chicago where the actual clock is.
Speaker 3 (34:42):
You know, it's cool.
Speaker 2 (34:44):
Where where are where? Where in fact are we actually today?
We were very close to midnight.
Speaker 4 (34:48):
I don't maybe we're ninety seconds or something or eighty
seven seconds, I forget. And I did have a conversation
with one of the AI CEOs who said he didn't
think the governments would regulate until there was a Chernobyl
scale disaster, and we haven't had that yet, so I
hope we don't have to have that in order to
(35:09):
have a regulatory regime. That's you know, corresponds to the
level of risk that the CEOs themselves.
Speaker 3 (35:18):
Are talking about.
Speaker 4 (35:18):
Right. They are saying, because a good chance will make
you all extinct. And you know, up to now, governments
have been saying, well, great, can we give you a subsidy,
can we streamline your permit process? And now maybe they're
realizing that actually, no, we need to protect the human.
Speaker 2 (35:36):
Race let's just talk about goals. The final question though,
I mean, so if you have the opportunity right now
to put the genie back in the bottle and live
in a world where there wasn't a computing excepent what
humans could do in their brains, is that the world
you would choose?
Speaker 4 (35:54):
I think no. I think there's a stopping point somewhere between.
And for example, I think you could have computers, but
just make sure that algorithms that are operating in computers
have to come with a proof that they are safe.
(36:19):
And this is actually an idea that dates back to
the nineteen nineties. Well proof carrying code, and you can
make computers that will check the proof of safety of
the algorithm before they run it.
Speaker 3 (36:34):
And so.
Speaker 4 (36:37):
If that becomes the standard for all hardware devices that
they only run checkable software objects that come with a
proof that it's okay to run this object, then I
think that would be a regime that would be extremely
hard to bypass because you'd have to have to create
(37:00):
a whole separate supply chain to produce you know, high
end chips. You know, talking about hundreds of billions of dollars,
tens of thousands of engineers decades of work that would
all have to happen, you know, in the black market.
Speaker 1 (37:14):
So to speak.
Speaker 4 (37:17):
So I do think there are these other stopping points
which are different from you know, just going right up
to the edge and hoping that no one.
Speaker 3 (37:27):
Chooses to go off, to go off the edge.
Speaker 1 (37:30):
It's to a Russell, thank you.
Speaker 3 (37:32):
It's been a pleasure.
Speaker 1 (37:44):
For text stuff I Musteloshian.
Speaker 2 (37:46):
This episode was produced by Eliza Dennis and Minister Slaughter.
Executive produced by me Julian Nutter and Kate Osborne for
Kaleidoscope and Katrina Novel for iHeart podcasts Our Engineer Today
was by Hate Fraser.
Speaker 1 (37:58):
Jack Insley mixed this episode. Kyle Murder wrote a theme song.
TechStuff News
Hosts And Creators
Oz Woloshyn
Karah Preiss
Popular Podcasts
Hey Jonas!
Hey Jonas! The official Jonas Brothers podcast. Hosted by Kevin, Joe, and Nick Jonas. It’s the Jonas Brothers you know... musicians, actors, and well, yes, brothers. Now, they’re sharing another side of themselves in the playful, intimate, and irreverent way only they can. Spend time with the Jonas Brothers here and stay a little bit longer for deep conversations like never before.
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by Audiochuck Media Company.
Betrayal Weekly
Betrayal Weekly is back for a new season. Every Thursday, Betrayal Weekly shares first-hand accounts of broken trust, shocking deceptions, and the trail of destruction they leave behind. Hosted by Andrea Gunning, this weekly ongoing series digs into real-life stories of betrayal and the aftermath. From stories of double lives to dark discoveries, these are cautionary tales and accounts of resilience against all odds. From the producers of the critically acclaimed Betrayal series, Betrayal Weekly drops new episodes every Thursday. If you would like to share your story, you can reach out to the Betrayal Team by emailing them at betrayalpod@gmail.com and follow us on Instagram at @betrayalpod and @glasspodcasts. Please join our Substack for additional exclusive content, curated book recommendations, and community discussions. Sign up FREE by clicking this link Beyond Betrayal Substack. Join our community dedicated to truth, resilience, and healing. Your voice matters! Be a part of our Betrayal journey on Substack.