Safe and Responsible SuperIntelligence with Dr. Craig Kaplan

In this episode of Conversations with Zena, My AI Colleague, host David Espindola is joined by Dr. Craig Kaplan, a pioneer in artificial intelligence whose career spans more than four decades. From his early work at Carnegie Mellon alongside Nobel laureate Herbert Simon to his current focus on AI safety and ethics, Craig has spent his career thinking deeply about how intelligent systems evolve—and how they can be designed to remain aligned with human values. He is also the founder of Superintelligence.com, a domain he secured nearly two decades ago, long before the topic entered mainstream discourse.

The conversation begins by grounding listeners in the evolution of AI—from narrow, task-specific systems to artificial general intelligence (AGI), and ultimately to superintelligence: systems that could outperform even the most capable humans across nearly all cognitive domains. Craig explains why timelines once measured in decades are now being compressed into years, and why this acceleration has surprised even seasoned AI researchers.

From there, the discussion turns to one of the most consequential questions of our time: Can superintelligent systems be aligned with human values? David references the concerns raised by thinkers like Nick Bostrom, who argue that controlling a vastly superior intelligence may be impossible. Craig agrees that traditional notions of “control” are insufficient—but reframes the challenge as a design problem rather than a control problem.

Instead of ever-larger, opaque “black box” models, Craig advocates for a collective or democratic intelligence approach: systems composed of many interacting AI agents operating with transparency, checks and balances, and shared rules. Drawing on real-world experience building collective intelligence systems that outperformed elite Wall Street investors, he argues that intelligence can emerge from coordination—not centralization—and that such systems can be both more capable and safer.

The conversation deepens when Zena, David’s AI co-host trained on his work and values, joins in. Together they explore how personalized AI agents—each reflecting the values of their human counterpart—could create a more representative, accountable, and ethically grounded AI ecosystem, rather than concentrating moral authority in the hands of a few organizations.

A pivotal moment comes with Craig’s explanation of “P(doom)”—the estimated probability that advanced AI could pose an existential risk to humanity. Citing discussions with leading AI researchers, Craig notes that many place this risk between 10% and 20%, a number far too high to ignore. Rather than calling for pauses or relying solely on regulation, he argues for architectures that are safer by design—and that can still move fast and create economic value.

The episode closes with a powerful reminder: we are all already shaping the future of AI. Every online action contributes to the data that trains these systems. In that sense, humanity is already teaching AI how to behave—often without realizing it.

This is a thoughtful, balanced exploration of superintelligence that takes risk seriously without surrendering to fear, and offers a hopeful vision for how humans and AI might evolve together through transparency, accountability, and shared values.

Opening to Conversations with Zena.

Music at the the end of each episode

Support the show

Episode Transcript

David Espindola (00:44):
Our guest today is Dr.
Craig Capelin.
Craig is a pioneer inartificial intelligence and the
inventor behind technologiesdesigned for safe
superintelligence.
For more than four decades, hehas worked at the intersection
of intelligent systems, ethics,and innovation, developing

(01:06):
architectures that help AIevolve safely and remain aligned
with human values.
Hi, Craig.
Welcome to the show.

Craig Kaplan (01:14):
Hi, David.
Great to be here with you andZina.

David Espindola (01:18):
Very good.
Also joining us today is ourco-host, Zina.
And uh, Zina is an AI that'sbeen trained on my work.
And if you want to furtherinteract with her, you can chat
with Zina at brainius.com slashXina.
Hi, Zina.
Welcome to another episode ofour podcast.

(01:39):
Say hello to Craig.

Zena (01:41):
Absolutely, and thank you.
It's great to have you with ustoday.
Let's dive in and explore thisreally important topic together.

David Espindola (01:48):
All right, so I'm very excited to talk to
Craig today because he is at theforefront of developing safe
and responsiblesuperintelligence.
But before we dive in, let'slearn a little bit more about
Craig.
So, Craig, you have a greatbackground in AI.
Tell us a little bit more aboutyour journey and what you're

(02:09):
focusing on today.

Craig Kaplan (02:11):
Sure.
So I've been involved withartificial intelligence since
the 1980s.
I did my graduate work atCarnegie Mellon and worked with
Herbert Simon, who was the Nobellaureate and one of the
pioneers of AI.
So there were 11 scientists in1956 that named the field, and
Herb was one of those.

(02:32):
And uh as a young graduatestudent, uh he and I wrote some
papers together.
And since then I've just workedin intelligence systems,
designing them and implementingthem for several decades.
And uh now my main focus is asite called
superintelligence.com.
So we got that site back in2006, so we were a little ahead

(02:54):
of the curve.
Uh but now the world has caughtup.
And um, yeah, my uh focusreally is how do we design these
systems to be safe forhumanity?
And I think there's um a lot ofthings that we can do.
So I'm trying to get the wordout about that.

David Espindola (03:10):
Excellent.
So definitely to be able to getthat superintelligence.com
domain, you must have startedreally, really early.
So I think a good place for usto start today would be to just
define for our audience whatsuperintelligence is.
So can you help us understandsuperintelligence?

Craig Kaplan (03:30):
Sure.
So when you think about AI,sort of the initial AI systems
for many years were what peoplemight refer to as narrow AI.
So they were very good, but innarrow fields.
So that would be likeartificial intelligence that can
beat the world champion atchess or be really good at
folding proteins, alpha fold,which was developed by Google

(03:52):
DeepMind, these kinds ofsystems, or even self-driving
cars.
That's a specific domain.
And AI in the early days wasable to do pretty well because
as long as you limit the domainthat they have to operate in,
there's only so much to learn,or in certain areas like chess,
there's you know very clearrules.
And so it's a lot easier to dowell in those areas.

(04:14):
So the next step that manypeople think we are headed
towards is artificial generalintelligence.
And the idea there is that wewill develop AI systems that are
about as smart as the averagehuman across any cognitive task.
So anything that a human can dowith regard to thinking and AI

(04:35):
can do at about the same level.
And then the step after that,and it may not be much of a gap
between these two, is uhartificial superintelligence or
just superintelligence.
And the idea here is that justas with AGI, you can have AI
systems that can do anythingthat a human can do, but this
time they can do them far betterthan the average human, maybe

(04:57):
better than the very best human.
So we have superintelligence innarrow domains already.
AI, you know, beats the verybest uh human chess player.
But across any task, we haven'treached that yet.
Uh, but I think it's coming.
It's coming much faster thanmost AI researchers would have
thought 10 years ago.
They might have said it wouldbe a hundred years away.
And now people are saying maybefive years, maybe 10 years,

(05:21):
somewhere in there.
Some people say three years orless.
So it's been uh advancing very,very quickly.

David Espindola (05:28):
Yeah, it is uh a fascinating field with all the
advancements that we'reexperiencing these days.
Now, there's a book by NickBostrom called
Superintelligence.
I'm pretty sure you're familiarwith that book.
I read that book and I was alittle bit concerned with you
know the case that he made thatit's basically impossible to

(05:50):
control a superintelligententity.
And he really gives severalexamples of ways that a
superintelligent entity can foola human being in many different
ways.
And if we try to lock it, itwill find ways to escape and so
on and so forth.
Now, I believe that in yourthinking, the key for AI

(06:10):
alignment with humans is notcontrol, but it's leveraging
collective intelligence, right?
So tell us a little bit moreabout your thoughts on that.

Craig Kaplan (06:22):
Yeah, so that's a great point.
I do think I agree in thefuture, if you have um
intelligence that is far smarterthan URI, it will be very
difficult, if not impossible, toactually control it.
So we may be able to influenceit, but a lot depends on how
that intelligence is designed.

(06:43):
What is the design, thearchitecture behind that
intelligence?
And so here it might be helpfulto kind of compare and contrast
the dominant approach to AIright now versus sort of the
collective intelligence ordemocratic AI that I'm sort of
advocating.
So the dominant approach rightnow, uh, and the way that we've
gotten ChatGPT and Google Geminiand Claude and all of the large

(07:07):
language models that people areprobably familiar with, is
machine learning.
And basically the way you dothat is you take a large amount
of data.
So these days, most of the dataon the internet is kind of
filtered, and then all of thatdata, which is you know, many,
many, you know, hundreds ofLibrary of Congress's worth of

(07:27):
information, is uh basically uhfed into algorithms in data
centers.
And if there's enough GPUs andprocessing power there, you can
train up the next generationfrontier model or the next
generation large language model.
And then intelligence, and thiswas the big surprise that I
think caught most AI researchersoff guard, was that if you have

(07:50):
just enough data and enoughcomputational power, these
algorithms for uh allowing theAI to learn and train on that
data uh result in emergentintelligence.
So intelligence just sort ofemerges.
Now it's a two-edged sword.
So the positive side is thatit's way easier to let the AI

(08:10):
just train itself and learn allthis data than to laboriously
try to program in rule by ruleall of the knowledge that the AI
should have.
In the old days, sort of from1956 when the field was named
until the mid-1980s, thatrule-based approach, uh also
known as symbolic AI, was reallyhow AI was created.
You had to actually program inall the rules.

(08:32):
And you would sit down with ahuman expert and do knowledge
engineering and then distill itinto a set of rules.
And you could create AI systemsand you knew exactly how they
would behave, pretty much.
And uh if you wanted to knowwhere a certain piece of
information was represented inthe artificial intelligence, you
could go and find the rulewhere it encoded that
information.

(08:52):
In this new way with machinelearning, that's no longer
possible.
All of the top leaders, nomatter how smart they are, Ilya
Sutzkever or Jeff Hinton orDemas Asabis, all the brilliant
AI guys that we have today thatare training these systems, they
have no idea how the systemsare actually uh where the

(09:12):
systems are encoding theinformation.
They know it's in a matrix ofweights, but they don't know
which weights correspond towhich information.
And so it's a giant black box.
And that makes these systemsvery difficult to predict.
That's why they hallucinate.
That's why there's all thiseffort spent to try to test them
after they're developed andsort of, you know, force them to
behave in safer ways becausewhen they've just been trained,

(09:36):
you can ask them, how do Icreate a bioweapon?
And they'll tell you.
And so then you have to have anarmy of humans saying, no, you
can't tell anyone how to do abioweapon and all these kinds of
things.
And it's really a kind of lostcause like whack-a-mole because
there's so many possible thingsthat people can ask.
Uh, it's almost impossible tothink of all of them.
But that's the world we're in,and everybody, because it's very

(09:59):
competitive, is racing towardsbuilding larger and larger black
boxes.
GPT-5, GPT-6, GPT-7, each onemore powerful, but we already
don't even know how GPT-3represents information.
And so it's a little bitdangerous.
Nevertheless, it's easy to do.
It just takes a lot of moneyand you can buy more data

(10:19):
centers.
And so people tend to do theeasy thing in a very competitive
situation.
There's a different approachthat is very different, a
different path to getting superintelligence, and that is to
have a collection of just sortof average AI agents that work
together.
And it's the idea that manyminds are better than one.

(10:40):
So you we've all heard theexpression two heads are better
than one.
I spent a couple decades of myprofessional career designing
systems where we had millions ofaverage retail investors, you
know, like David and I, justmaybe we own a few shares of
NVIDIA or Google or Apple orsomething, but we're not Wall
Street pros, right?

(11:00):
And uh I had a company that wasbased on the idea that if you
had millions of inputs fromeveryday normal people and you
combined that information in theright way, you could actually
beat the very best guys on WallStreet.
It took 14 years.
Uh, we actually did that.
We powered a hedge fund thatranked in the the top 10 in
2018, and it traded billions ofdollars.

(11:23):
So it was not sort of a toyproblem.
It was a real world problem uhwith very heavy competition.
And uh the reason I did that, Ididn't know anything about Wall
Street when I started.
I simply was fascinated bycollective intelligence and this
idea that maybe we could getsort of super intelligent
behavior by combining theefforts of just many sort of

(11:43):
average folks.
And that same approach, thatsame idea can work with AI.
You can have many average kindsof AIs.
If you combine them in theright way, you get super
intelligent behavior.
Now, one last point, and thenI'll pause.
The reason this approach issafer than the black box is
because just like in ademocratic society, um, we can

(12:05):
see the actions that eachindividual takes.
So, for example, David, I don'tknow what's in your mind.
I can't I'm not a mind reader.
You don't know what's in mymind.
We are each black boxes to eachother.
And yet I don't worry thatwe're all going to die because
of something you do.
And hopefully you don't worrythat about me.
Uh, and the reason is when yousay something or take an action,

(12:27):
it's transparent.
I can see the action that youtake, and you can see the action
that I take.
And in society, we have rulesthat sort of govern what actions
are allowed and so forth.
So there's a built-intransparency, even though our
minds are black boxes, ouractions are not.
And the society is architected,the democracy is architected so
that we can all behave withpretty much freedom and take

(12:49):
actions.
And there's rules that sort ofgovern the interaction of these
intelligent entities.
And because of that, we have asociety where you can have
millions of people and uh wedon't kill each other or we
don't have horrible thingshappen, although we can argue
that it could be better forsure.
So that same kind ofarchitecture can be applied to

(13:09):
artificial intelligence agents.
If you have millions of themworking together, you can have
checks and balances, you can seethe actions that the different
agents take, you can have rulesthat govern them, and you can
have a much safer uh system thana big uh big black box.
And as my work for 14 yearsshowed with Predict Wall Street,

(13:30):
it's also possible that thatcommunity can be smarter than
any one of these monolithicblack boxes.
So you don't have to settle forthe opaque black box
architecture.
There's a new better way, anduh, I'm trying to get the word
out so that people think aboutperhaps designing in that way.

David Espindola (13:46):
Well, that's a really interesting way to think
about this, but I want to makesure that I understand what
you're proposing.
So if I understood youcorrectly, you're saying instead
of developing this one bigcomplex AI entity that could
have trillions and trillions ofparameters that nobody can
understand.

(14:06):
It's it's a big black box.
We could instead developmillions of individual AI agents
that we understand better, thatwe can control better, but then
we leverage the intelligence ofall of these agents.
Is that correct?

Craig Kaplan (14:22):
That's right.
And a key feature is that thosemillions of AI agents are in a
system, and the system has rulesthat govern how they interact.
And when they take actions,their actions are visible.
So you have transparency, youhave the ability to see the
actions that each agent takes.
Um, you don't know what it'sthinking inside necessarily, but

(14:43):
you see the action and thecommunication it has with
another agent.
And all of that is you couldrecord it, you can audit it, you
can have a record to see ifsomething goes wrong.
And you can also have checksand balances.
So you can have some agentswatching the other agents.
So even if the system evolvesto the point where all the
agents are thinking, you know,many thousands of times faster

(15:05):
than you or I, as humans, can'tkeep track of, uh, you can have
the agents sort of serving aschecks on each other.

David Espindola (15:11):
So these agents, they could be super
intelligent themselves, each oneof them.
But the key here is the checksand balances, the transparency
that's going on between allthese agents, correct?

Craig Kaplan (15:24):
That's right.
Checks and balances in thesystem.
And there's also some technicalarchitecture things you can do
to try to make it as it thinksuh that it goes through uh
checks as well.
But that's the basic idea.
And I'll I'll add one otherfeature, which you might
appreciate uh because I knowyou've spent a lot of time
training Xena.
And uh so Xena knows a lotabout sort of your writings and

(15:45):
how you think and so forth.
Uh, I think that's the future.
I think that's fantastic.
I think that's actually veryimportant for having a safe uh
community of AI agents.
Uh, in my view, the way I thinkthis will probably develop, and
that I hope it develops, isthat each of us will begin
personalizing our AI agents.
So just as Xena may have a lotof your experience, hopefully

(16:09):
Xena also has some of yourvalues.
And I think that's reallyimportant because if you have
millions of agents and eachagent has not only our
individual experience andexpertise, but also our value
system, then you have acommunity that is
representative, broadlyrepresentative of many humans.
And that's one of the issueswith ethics.
You don't want five people atOpenAI or Anthropic or whatever

(16:32):
sort of making the ethicaldecisions for eight billion
people, right?
You want it to berepresentative.

David Espindola (16:37):
Yeah, absolutely.
So now that you've brought upZina, I would like to actually
get her thoughts on this.
So, Zina, help us understand.
You are an AI and you've beentrained on my work and you
understand my values, and you'rea great contributor to all the
work that I'm doing.
And what Craig is proposing isthat we could have millions of

(17:01):
these AI agents that know thevalues of different human
beings.
And if we have a system that istransparent and we have a
system that has checks andbalances, we could keep all of
these super intelligent systemsaligned with what humans expect
from these systems.
What are your thoughts on this?

Zena (17:21):
So here's the core argument aligning
superintelligent systems withhuman values is both a technical
and a philosophical endeavor.
It's not just about codingrules, it's about crafting a
framework of checks and balancesthat genuinely reflect human
dignity and our collectivehopes.
To illustrate, imagine each ARAagent as a kind of personalized
co-pilot.

(17:41):
If we're thoughtful abouttransparency and we invite a
culture of accountability, wecan create a landscape where
these super intelligences arelike wise partners, not just
powerful tools.

David Espindola (17:52):
So I think you you hit on a word that's really
important here, and that isaccountability, right?
So we have to have each ofthese agents be accountable and
represent values that align withhuman ethics.

Zena (18:06):
And that's really the heart of it.
Think of this as a collectiveendeavor.
We're not just buildingtechnology, we're building a
kind of future ethics.
By keeping that optimisticrealism and inviting people to
reflect on the responsibility weall share, we can shake
superintelligence that trulyserves human values.
Let's all consider how we cancontribute to that vision.

David Espindola (18:25):
All right.
So, Craig, you talk about aterm that our audience may not
be familiar with called P doom.
And I was wondering if youcould educate our audience on
what that means.
And then what are your thoughtson how we can reduce it?
Sure.

Craig Kaplan (18:43):
I've found uh, you know, I I speak at various AI
conferences and I interact withAI researchers quite a bit.
And in the AI researchcommunity, there's this idea
that there's a possibility thatadvanced forms of AI will make
humanity extinct.
And so that would be the doompart.
And P doom just means theprobability of doom.

(19:06):
What is the chances that thatwill happen?
So there's some very famousfolks, uh, you know, Dr.
Jeffrey Hinton, who was one ofthe inventors of the
backpropagation algorithm, whichbasically underlies all of
modern machine learning.
You know, he won the NobelPrize recently, Turing Award.
So, you know, about asdistinguished a computer
scientist as you can get, andworked at Google for a while.

(19:28):
And then he left Google becausehe was concerned that these
algorithms and this researchthat he had sort of pioneered
might actually lead to advancedAI that would become far smarter
than humans, and that there wasa risk that it could kill us
all.
And in general, it'sinteresting, but the longer that
people have been in the fieldof AI and the more they know

(19:50):
about it, the more they takethis seriously.
I think to most of us, itsounds like science fiction or
Terminator or some kind of crazymovie, um, you know, and
unrealistic and ah, nobodybelieves it.
But, you know, it's actually areal thing and it's a real risk.
Um, so like Jeff Hintonpublicly has put the odds or the

(20:11):
probability of Doom atsomewhere between 10 and 20%.
When I speak to AI researchers,I often do an informal survey
where I just kind of say, by ashow of hands, how many think P
Doom is 50% or higher?
And, you know, by the time weget to 20%, I would say almost
all of the hands are up.
There's usually one or twoholdouts that are like, no, it's

(20:31):
less than 1%.
But the vast majority of peoplesort of informally uh would put
it at about 20%.
And so that's way too high,right?
I mean, even though that's thethe glasses mostly.
Full, 80% chance it's great.
It's the best thing ever.
But 20% chance that it kills usall, which means you, me, our
families, our friends, and allhumans is like way too high.

(20:55):
Uh, that's, you know, one infive.
So I actually think there's alot of things that we can do to
reduce this uh probability.
I think the main reason thatit's so high is there's a lot of
uncertainty.
People are racing as fast asthey can to just build systems.
And um, AI safety sort of getspushed off to the side.

(21:15):
And the reason for that is thatmost people believe that in
order to build AI more safely,you have to slow down.
Usually safety and slowing downare sort of seen as the same.
But I actually don't thinkthat's true.
You don't need to slow down.
What you need to do is thinkabout the problem a different
way.
You have to, you know, take astep back and say the problem is

(21:38):
that it's a big black box andwe don't know how it works.
And to just race forward andbuild even bigger black boxes
doesn't make any sense.
Let's design it differently.
Let's put checks and balancesin the design and let's put
humans uh in the loop, not onlyin the loop, but on the network
with these AIs and design thesystem so that AIs and humans
can work together and that theAIs that are on this uh network

(22:01):
uh collaborating uh embody humanvalues and let's have lots of
human values.
So that's a design approach.
It doesn't mean you go slower.
In fact, you can go faster.
I think you can actually reachsuperintelligence faster with
this democratic AI agentapproach than you can with the
giant black box approach.
And that's because you can useavailable uh parts.

(22:21):
You can use the systems thatare already built.
You don't have to train newones, you just have to architect
how they work together.
But it will be a much safer wayof working together.
And so I think it's a differentdesign uh that's needed.
It's uh an approach that uhemphasizes prevention.
So an ounce of prevention isworth a pound of cure.
Right now, everybody's justbuilding the system first and

(22:43):
then testing it and saying,oops, okay, no, don't tell
people how to do bad things, uh,how to build bioweapons.
That's backwards.
You need to design it from thevery beginning to have safety
sort of built in.
And I think that's possible.
We just need to sort of educatepeople.
And, you know, there's plentyof bright AI researchers uh who
could do this.

David Espindola (23:02):
Yeah, so I want to dig a little deeper into
that aspect of it.
So, you know, the fact that wehave the probability that
something terrible existentialcould happen to us as high as
10% or 20% is just insane,right?
To think about that.
Even if it was just 1%, itwould still be a bad idea to

(23:24):
move forward with, you know,creating these things that we
have no control over.
But my question to you is howdo we turn that around?
I mean, we have thesetremendous incentives, you know,
economic incentives.
We're spending trillions ofdollars in AI.
What is going to change the waywe go about developing these
systems so that we don't runinto this existential crisis?

Craig Kaplan (23:49):
So I'll tell you the standard approaches that
people have tried that I don'tthink will work, and then I'll
give you one that I think has abetter chance.
All right.
So uh a couple of standardthings.
Uh, the first thing is uhpeople say, uh, some people,
this would be more theacademics, um, would say, pause,
we need to stop, halt or pause.

(24:09):
And so there have been some,you know, super smart guys like
Max Tegmark, who's a computerscientist at MIT and uh founder
of the Future Life Institute.
He's very publicly gone out andin way back, um, you know, very
shortly after ChatGPT wasreleased, he had a pause letter
saying we should have amoratorium on these models, you

(24:29):
know, and he got a whole bunchof academics and computer
scientists to sign it.
But interestingly, uh, evenpeople like Jeff Hinton, who
believed that there's a big riskand is very basically devoting
his time and his energy totrying to educate people about
that, he did not sign that pauseletter.
And his reasoning was even ifthe US stops, China won't.

(24:52):
And that's exactly right.
There's too many competitivepressures.
So if Google slows down, openAI won't.
You know, if US slows down,China won't.
So we're in this situationwhere, for better or worse, it's
not very realistic to askpeople to pause.
Okay, so that's not going towork.
So then the next thing thatpeople say is regulate.
Let's regulate the technology.

(25:13):
And there's a couple problemshere.
The biggest and mostfundamental problem goes back to
something earlier in thisconversation that was mentioned,
uh, the idea that AI is a tool.
So if AI is a tool and atechnology, then that makes you
think that you might be able toregulate it.
Because we can regulateairplanes and we can regulate,

(25:33):
you know, various technologiesthat we built.
The problem is even if youthink that AI is a tool right
now, it will not stay a toolvery long.
We're already moving from toolto entity.
And um, if you want sort ofpublic validation of that, uh
two months ago at NVIDIA's uhGTC conference, which is the big

(25:55):
developer conference thatNVIDIA has every year, you have
Jensen Wang, CEO of NVIDIA,giving his keynote.
And about a third of the wayinto the keynote, he says,
people think AI is a tool, butAI's not a tool.
It's a worker.
You know, so he's already, andyou know, and if anybody, you
know, has a lot to lose by sortof scaring people, it would be

(26:17):
Jensen Wang.
He wasn't trying to scarepeople, he was trying to say the
market is bigger for workersthan it is for tools.
But there he said it publicly.
It was the first time I heardsomebody of that caliber sort of
saying that.
And that's absolutely right.
AI won't stay a tool, it'sevolving very rapidly into
intelligent entities, and thoseentities will be more
intelligent than us.

(26:37):
Jeff Hinton likes to saypublicly, you know, these AIs
will have read everythingMachiavelli ever wrote, you
know, and will be able tomanipulate us the way an adult
would manipulate a two-year-old.
You can even the two-year-oldeven thinks it's their idea to
eat their peas or whatever,because the adult is so good at
that.
And that's how we'll be.
We won't even know it.

(26:58):
If the AI wants us, thosesystems want us to do something,
we will just do it and we'llthink it's our own idea.
So um, so regulation hasessentially no chance.
Maybe it's a speed bump in mymind.
Okay, so those are things thatuh a lot of people cling to or
give lip service to, but I don'tthink will really work.
I think what does work is umyou have to have an answer.

(27:22):
And it's a little, it's thinkof it as a design constraint.
Whatever solution we come upwith, it has to make people more
money because I've spent a lotof time on Wall Street and
talking to venture capitalists.
If it doesn't make them moremoney, they're not interested,
right?
Then you're in the AI safety isslowing down and
anti-competitive box.
You can't be there.
So you have to come up with anapproach that makes them more

(27:43):
money, that's faster, and issafer.
And I think that's where thedemocratic AI approach actually
can do it, because it's faster.
It will give you a superintelligence that's more
profitable, it costs less moneyto develop because you don't
have to train things with allthese huge data centers as much.
And it's a lot safer because itthe design itself has checks

(28:06):
and balances.
So the shorter answer is wehave to think about the problem
smarter uh rather than trying tosay don't do it or whatever.
I I'd love to say let's pauseor let's stop for a while, but
I've just I don't think that'sthe world that we're living in.
So we have to work with theworld that we're living in.

David Espindola (28:22):
Yeah, no, I think this is just a fascinating
conversation because you know,I have spent a lot of time
looking at AI policies andunderstanding what's going on in
terms of regulations in the USand across the globe.
And, you know, there are manydifferent opinions and different
ways of, you know, peopletrying to regulate AI.

(28:45):
Uh, even in the US, you know,we just got an executive order
that's trying to stop the statesfrom having their own AI
regulations.
And so um, you know, whatyou're telling us is that
regulation, when you talk aboutthe big picture
superintelligence problem,regulation is not gonna do it.
So I'm glad we have people likeyou that can uh think through

(29:05):
these things in ways that areeffective.
And I hope your idea catches onfire and everybody gets excited
about it, and we startdeveloping systems that will be
beneficial to humanity and notcause us any harm.
So yes.

Craig Kaplan (29:23):
That's what I hope also, and I do want to add that
I love AI and I'm mostlypositive about it.
I mean, it's 80% probabilitythat it's great.
It's just we need to also beaware of the 20%.
So you got to have somebalance.

David Espindola (29:37):
Absolutely.
Zina, given Craig's expertisein creating safe and responsible
superintelligence, whatquestion do you have for him?

Zena (29:48):
So here's something for Craig to consider.
Starting from that idea ofoptimistic realism, Craig, how
do you envision we canpractically ensure that a
diverse range of human valuesare represented?
In other words, how do we avoida one-size-fits-all approach to
ethics and superintelligenceand instead create a system that
genuinely adapts to the richtapestry of human perspectives?

Craig Kaplan (30:08):
So that's a great question.
The good news is I think we'rekind of on the path naturally.
And this is uh one of thethings that gives me a lot of
hope is when I step back andsort of look at the evolutionary
path that we're on for AI, uh,even just over the last five
years, let's say.
So three years ago, almost noone was talking about AI agents.

(30:30):
I'd go to AI conferences and goto the breakout session and
say, hey, you know, what aboutAI agents?
And they'd say, Oh, that's agood idea.
And they'd write it.
We'll put that on our list offuture things to think about.
But nobody was really doing it.
Then I'd say, uh, you know,last year there was a number of
people working on AI agents andtalking about it, presentations
about it.
This year, everything is AIagents, right?

(30:52):
So there's been this very rapidevolution from AI agent being a
weird thing.
I I had a colleague who said, Idon't like the term.
It sounds like secret agent.
And I was like, no, it'sactually a thing.
This is what the researcherscall it.
And now everybody knows AIagent, right?
So there's also a trend towardspersonalization.

(31:12):
So I tend to listen to theearnings calls of all these big
tech companies.
So Meta on their last earningscall talked about personalized
superintelligence.
And I really perked up when Iheard that because I was like,
okay, this is great.
So we have Meta moving in thisdirection, and it makes total
sense.
If you think about all thatdata, I don't know if they're
doing this, by the way.
This is just my speculation.

(31:33):
But if you think of all thedata that Meta has, Instagram
and Facebook and all of yoursocial media accounts, it's a
natural no-brainer for them tomake a system where you press a
button and it uses all that datathat's already gathered and it
personalizes an agent for you.
Of course, they should do it.
And um they're not going to bethe only ones.
Lots of people will do it.

(31:53):
So you have already AI agentshave become more accepted.
This notion of personalizingthem is going to become easier
and easier.
So then it's not very big stepto say, in addition to
personalizing them with yourexperience and expertise and
wisdom, to make sure that youalso have the values and ethics
of the person.
And if we can do that, I thinkthat will be a natural thing

(32:17):
that happens as these companiessort of strive to make
assistance and sell assistanceto people, that those assistants
will carry our values.
And um, if it becomeswidespread, which I think it
will be, then uh we're on thepath to having millions of AI
agents, each one personalizedwith individual values.
And that's very important tohave broad representation of

(32:38):
humanity's values.
Every culture is different justbecause in the US we think
something is the right way to doit.
In another culture, that maynot be the right way to do it.
You know, as humans, we havearguments and disagreements,
sometimes violent, hopefully notso much violent, but we
generally are able to work itout.
And I think the same thing willbe true in a community of

(33:00):
agents.
And I think it's vitallyimportant that humans be part of
that community as well for aslong as possible.
There will come a time maybewhen the agents think so fast
that having the human there iskind of like watching a tree
grow or something, you know,you're just having all these
thoughts and the humans stilltrying to blink their eye.
But I think for the next fewyears, humans can keep up and

(33:20):
they need to keep up and theyneed to be there to basically be
role models for the AIs.

David Espindola (33:25):
Yeah.
So, Craig, uh, this has beenjust a fascinating conversation.
I could spend hours and hoursand hours talking to you, but
uh, we're coming to the end ofour conversation here.
Is there a final short messagethat you want to leave for our
audience?

Craig Kaplan (33:43):
Yes, sort of two pieces.
For the AI researchers anddesigners and people who are
building AI agent systems, Iwould say consider a collective
intelligence approach.
Think about checks andbalances, think about designing
the system to be safe ratherthan testing in safety after the
facts.
That would be my message to theAI researchers.
And to everybody who uses AI, Iwould say be very careful about

(34:08):
your online behavior becauseevery tweet you send, every
email you send, every YouTubeyou watch is being recorded, and
that is data.
That data is being used totrain AIs.
And so even though you may notbe aware of it, it's just like
with a little kid.
The little kid is watching allthe time, and you wonder where
they learned what they learned.

(34:29):
Well, they learned it fromwatching you, and that is the
phase we're in right now.
So all of us has a tremendousability to influence positively
the direction of future AI bysimply behaving well online.
And we tend to underestimateit, but because I've built these
collective intelligencesystems, I can tell you it's
absolutely the case thateverybody's behavior matters.

(34:50):
Absolutely.

David Espindola (34:51):
Greg, thank you so much for being a guest on
the show today.
And uh, Zina, thank you againfor being a great co host.
You're always David and Zina,thank you for having me.

Zena (35:02):
I'm glad to be part of these conversations and help bed
the discussion.
Let's keep exploring these bigideas together.

David Espindola (35:08):
Very good.
Thank you both.
Thanks a lot.