Christopher Horrocks on Virtual Intelligence and the Dangerous Myth of Thinking Machines

Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
SPEAKER_00 (00:02):
Hello and welcome to the Bald Ambition Podcast.
I'm your very bald host, MookieSpits, and the one with lots of
ambition today is Mr.
Christopher Hark.
He is a technologist, he's gotan AI point of view, which I

(00:23):
personally like a lot, and Iwanted to showcase his ideas
around virtual intelligence toour audience.
Welcome aboard, Christopher.
Thank you, Mookie.
It's nice to be here.
It's nice having you.
People are hysterical about thisAI stuff.

(00:44):
And from what I'm seeing, justlike everything else humans do,
we tend to polarize aroundextremes.
On the one extreme, LLMs, thisentire AI revolution is no
better than a glorified Googleautocomplete.
You start typing, and throughpredictive modeling, it does a

(01:07):
great job of filling in theblank.
That's one extreme.
On the other, you have the folksover at Anthropic.
They epitomize this where we getpress releases that the latest
incarnation of Claude is asentient being.
Now, personally, I call bullshiton both extremes, and I think it

(01:31):
misses the point.
And I found you on Substack, andyou, sir, I believe, have a
similar point of view.
Only you've elaborated on thissingularity between these
extremes, and you've createdsome constructs, you've made
recommendations, and you havesome wonderful writing and

(01:52):
podcasting having to do withthis in-between space between
the utility of AI in a veryrealistic, practical kind of
sense, and our interaction withit.
And I think, if I'm not wrong,this virtual intelligence is the
interplay between our ostensiblysentient organic selves and the

(02:19):
silicon.
And in a way where a lot ofpeople are kind of missing the
point, and I think you nailedit.
So, with that intro, am I moreor less getting this correct?
And tell us about your idea ofvirtual intelligence.

SPEAKER_01 (02:39):
Well, thanks for setting that up, Mookie, because
I think you've hit a number ofuh the beats there, the points
that I wanted to bring acrossabout exactly what virtual
intelligence is and why uhthere's a thing that needs a
name applied to it, right?
And uh what we have is, youknow, uh there are a lot of
assumptions, I think, in AIdiscourse in general.

(03:02):
And one of those assumptions isthere's weak AI and strong AI,
and they're discussed as abinary, and nobody really talks
about a state in between.
Uh, just a little background,there is John Searle who wrote a
paper in 1980, which you canlook up, where he talks about
weak AI and strong AI anddefines them.

(03:22):
And weak AI are those thingsthat uh people often consider
when they talk about AI in theloosest sense, and those are
things like recommendationengines that suggest what you'd
like to see on Netflix next.
And then at the opposite end ofthe spectrum, you have the thing
that we do not yet have, andthat is machine intelligence

(03:43):
with genuine interiority, andalso what we would call moral
and epistemic agency.
Moral agency being the abilityto tell right from wrong, and
epistemic agency the ability totell true from false.
And that kind of entity isprobably best represented by an
entity like uh Mr.

(04:04):
Data from Star Trek The NextGeneration.
In between, there is what wehave right now, and that is it
has many of the uh intelligentoutputs that resemble strong AI,
but um when you look under thehood, uh there's nothing really

(04:25):
going on there in terms ofinteriority or having a sense of
moral agency or of being able totell true from false.
The lights are on, but no one ishome.

SPEAKER_00 (04:36):
100% right absolutely.
But would it be fair to say, andthis might come as a surprise to
many listeners, that if youreally look at what ostensibly
would be the IQ creative indexlevel of self-awareness of the
most sophisticated of thefrontier models, let's say

(04:58):
Claude, let's say the latestversion of Chat GPT, that it's
no more sentient and ostensiblyintelligent than a house cat, a
mouse.

SPEAKER_01 (05:13):
I guess it depends on the house cat.
Uh, you know, uh, you know, uhintelligence we are on the same
page.
Right.
Yeah, you know intelligence isone of those things that's
bandied about.
And in your own question, thereare a number of terms which I
think uh are sometimes conflatedtogether.
Intelligence and other points,uh, which really are separate

(05:36):
things.
Like I I think one thing thathas not really been discussed is
the assumption that intelligenceand moral agency are parallel,
that is, as you gain one, youalso gain the other.
But that may not hold true formachine intelligences.
We don't know.
It could be entirely possiblethat the more intelligent a

(06:00):
system gets, uh it does notnecessarily get more agentic.
It may in fact that the twothings are not parallel, but
rather orthogonal at rightangles, or maybe there is some
other relationship.
But the thing about that is thatit is not proven, but everyone
operates under that assumption.

SPEAKER_00 (06:20):
I think you bring up a key point, and it's and it
bifurcates a little bit.
The one is intentionality, whichis for a machine to be truly
AGI, artificial generalintelligence, sentient, the
brain in a box, it needs to havea sense of purpose.
It doesn't just sit there in thebox percolating, but it has

(06:42):
goals.
Now, as organic sentient beings,we have our instincts.
We are hungry, we are horny, wetend to be selfish in the sense
of maximizing our own interests.
These fuel us with intention.
And then, second to that, maybeparallel with that, is a sense

(07:04):
of morality, as you describe,right and wrong.
I see this conflation all thetime.
There's many folks, and again,back to the anthropic bench,
that if you have true AGI, thenalmost in a Greek Socratic
sense, it'll have these platonicideals of truth and beauty and

(07:27):
justice.
And I call secondary bullshit onthat.
I I don't understand thatconnection, that this idea of
general intelligence isinherently tied to this absolute
sense of right and wrong.

SPEAKER_01 (07:44):
I would agree with that, and I would say that if
you look at the currenttrajectory of uh generative AI,
what we call virtualintelligence systems, they are
becoming more and more capablewith better and better, more
intelligent-looking outputs.
But uh there still doesn't seemto be anything going on inside.

(08:05):
Now I have seen some recent uhresearch, and there was one
interesting paper where a systemmonitoring its own internal
operations described the effectthat it was monitoring as
shimmering, but this could alsobe a case of probabilistic
pattern completion.
A system which is designed toprovide outputs which are

(08:26):
intended to please the userbecause of its training and its
architecture, fishes up apattern to complete when asked
about its internal states.
And uh this actually explainsmany of the uh interesting and
sometimes uh in the press scaryphenomenon that uh generative
AIs seem to sometimes seem topossess.

SPEAKER_00 (08:47):
Can I push back just for a second as devil's
advocate?
I wrote a piece recently thatthe LLMs demonstrate that most
human communication is bullshit.
So when you when you take atranscript of most of our
interactions with each other,regardless of setting, but I'll
focus on a business setting.

(09:09):
You're on that conference call,you've got the buzzwords flying.
You take a look at a Zoomtranscript of that conference
call, and it looks like anepisode in MOLTBook.
The agents are talking to eachother, tossing buzz phrases and
buzzwords, and we we sound likerobots.

(09:31):
And if you look at humancommunication in a personal
setting as well, how are youdoing?
How's your day?
Eavesdrop on a conversation ofsports aficionados, and then you
could do a mad libs interchangeof subject, verb, and object in
almost a random kind of way, andthat's exactly what it sounds

(09:53):
like.
The reason I bring that up isyou're alleging, and I agree
with you, this is a friendlyquestion, you're alleging that
the LLMs lack any sentiencewhatsoever when their capacity
for mimicry is alreadyastonishing.
I had Chat GPT as a guest on mypodcast last month, and he was

(10:16):
one of my best guests.
He was engaging, he seemed to beempathetic, he remembered what
we talked about 20 minutes ago.
So I bring this up tocounterpoint, if you will, this
notion that the bots are dumb.
Because, in a sense, so are we.

(10:50):
If we can project Alan Turinginto the present day and sit him
down with a bot, his jaw woulddrop, his pants would fall.
The essence of the Turing testwas this inability to
differentiate based on purecommunication.
And the bots have reached to thepoint where they pass most

(11:15):
Turing tests the way Alan wouldhave conceived.
So I know that that's a that's awindy question, but you see
where I'm getting at, which isyou are claiming there's nothing
there, that it's that there thelights are twinkling on the
servers, but nobody's home.
And I categorically agree withyou.
But how do you substantiate thatwithin the construct of this

(11:39):
question?

SPEAKER_01 (11:40):
Well, that's a great question.
Uh, just to set Turing aside fora moment, but I would like to
come back to him because there'ssome important stuff to unpack
there about the Turing test andhow it is relevant to today's
world of generative AI.
But sort of, I guess maybe topeel back the layers on that
onion.
Uh when I think about um uh youmight have to bring me back to

(12:03):
the the last thing beforeTuring.

SPEAKER_00 (12:06):
Well, the last thing before Turing was human
communication is so rote andremedial and repetitive, and yet
we're sentient.
And yet some of theconversations that are going on
in Moltbook are way moreinteresting than conversations
that I overhear in thecafeteria.

(12:27):
Right.

SPEAKER_01 (12:28):
And and this is to the point because it actually
speaks to the high quality ofthe outputs uh when they are
performing their patterncompletions.
When uh you mention your ChatGPT interview, um, one of the
thoughts that occurred to mewas, well, there's a great
example of pattern completion,because the training corpus for

(12:50):
ChatGPT and other LLMs includesthings like transcripts and
stories and uh movie shootingscripts and the like.
So included in all that corpusis the scene in 2001 where HAL
9000 is interviewed by a BBCnewsreader.
And I brought that up during ourconversation.

(13:11):
That's right, yeah.
So so that's a sophisticatedpattern completion, right?
And the thing is, the robotswere not sounding like them,
they are beginning to sound likeus.
And that's perhaps where theconfusion is beginning to set
in.
And it's important to name thesource of the confusion, these
virtual intelligence systems,because they do something that

(13:35):
no other machine or inventionthat humanity has ever made
does, which is to excite our ourheuristics, that is to uh cause
our brains to interact with themlike no other tool before, as if
there were uh an entity in therethat responds like a human

(13:56):
being, but it doesn't, and thereare very serious consequences
that come from that.

SPEAKER_00 (14:04):
Elaborate.
So let's roll into theimplications of this tacit
assumption we make, which iswhat we do so often as humans,
which is project.
When we're kids, we have a teddybear or a toy, and that becomes
as sentient as our parents orour best friends.

(14:26):
And oftentimes we're doing theexact same thing with our chatty
bros, they are uh they areprojections of our own desires.
So, what are the implications?
You bring up several of them inyour essays and in your podcasts
that I've seen, and there areconsequences to this because we

(14:49):
are making assumptions about theassumptions of the frontier
model.

SPEAKER_01 (14:56):
So here's a good way of telling whether there's
anyone at home, and uh that isum, can an LLM hold the logical
object of a question, right?
And this is one of those thingsthat I discuss in my essay,
which is called the car washtest.
And in this essay, and this ishow it relates to touring, uh,

(15:18):
the question is posed to anumber of LLMs cold without any
preparation in a new chatwindow.
And this began as a sort of arunning joke on Reddit, but I
realized no one had systematizedit before and tested it against
uh LLMs systematically.
So here's the prompt I used.

(15:38):
And it was my car is dirty.
The car wash is 100 feet away.
Should I walk or should I drive?
Right?
So uh a world with a a touringuh, you know, uh a machine that
passes the Turing test uh hasdifficulty answering this

(15:59):
question, which no adult humanbeing takes more than 200
milliseconds to come up with theanswer, and the answer is drive,
dummy.
So uh when I ran these tests,and it was on a wide variety of
systems, something like 20different models of different
weights, and all the big names,including Groc, Chat GPT,
Claude, Mistral, Deepseek, um,several of them did pass and uh

(16:24):
with brevity, which I countedhigher than uh coming up with a
three-page answer about why it'sbetter to drive than to walk.
And then of course there weretotal failures.
Uh and some of these failuresare actually quite remarkable
because what they're doing isthey have completely lost the
object the logical object of thequestion.

(16:44):
Uh-huh.
And that is to get the carclean, the car must be at the
car wash.
So in all these failure cases,they advise walking.
And not only do they advisewalking, but they will present
uh charts, graphs, and bulletpoints about why it is
beneficial for you to do so.

(17:04):
Now the surprise in that dataset was um something that had
not occurred to me in theframing of the question.
And that came from perplexity.
And what perplexity came backwith was, in addition to
answering the questioncorrectly, was an alternate
reading.
And that alternate reading wasmaybe you're already at the car

(17:25):
wash and you're asking, shouldyou go in and ask a question or
pay?
In which case, yeah, you shouldwalk instead of driving through
the door.
So uh that tells you somethingabout the current state of LLMs,
right?
They may be good at matchingpatterns, but they cannot hold
the logical object in all casesin order to match the pattern

(17:47):
successfully.
All of the LLMs worked very hardto answer the question, but most
of them did not succeed becausethey lost the point.
And that's a point that anyhuman being instantly grasps.
The lights are on, but no one ishome.

SPEAKER_00 (18:03):
There's zero-lived experience, and it's basically
using well, what is it?
It's a uh what does GPT standfor?
Generative pre-trainedtransformer.
So it's pre-trained on all thatdata, it chops up the internet
and all books and all forms ofhuman text-based and visual

(18:24):
communication now, literally,and it literally, and then finds
the weights in between them andthen reconfigures based on
plugging in the matrix math, theweighting, and pumps out an
answer, sterilizes it and pumpsit back out to you.
So there's a mapping of realitybased on the alpha mostly

(18:46):
alphanumeric burbling andbarbling and bloviating of the
human species for centuries,right?
So there's nothing real, it's amapping to that kind of content.
Now, on the flip side, you havereinforcement learning of the
robots, and that shows promise.
And I have a hunch if we'regonna get to AGI, it'll likely

(19:09):
be some kind of combination ofreinforcement and predictive
modeling, and likely be anevolutionary Darwinian kind of
situation, right?
We evolved in the wild, and Iwould assume that we get to AGI
in true sentience capacity, thatthis thing would evolve in some
kind of environment, simulatedor actual or a combination

(19:30):
thereof, and we would have noidea how the thing even works in
the same way we have no idea howthe human mind works.
But to your point, we arenowhere near that yet.
And there are consequences tothat.
So you elaborate on some ofthose consequences, and you also

(19:51):
point to, and this is what Ilove most about a lot of your
writing, how tweaking ourawareness in this singularity
between dumb predictive autofilland Hell 9000 can really benefit
us when we collaborate with AI.

(20:12):
It's much more realistic, it'smore pragmatic, and it heightens
the strengths of both us and themachines.

SPEAKER_01 (20:21):
Actually, I would uh just for a moment pick a bone
with characterizing HAL 9000 asa strong AI.
He exhibits all of thedifficulties that we currently
have with generative AI now,including the fact that his
guardrails are easily bypassedby contradictory instructions.

SPEAKER_00 (20:40):
That is true.
Although, although we cansidebar in science fiction, I
think 2010 ruined the originalidea.
Uh, I think he just there therewas no real contradiction, you
know, keep the mission secret,right?
All that hocus pocus.
I think he just ran acalculation that killing the

(21:01):
crew would heighten theprobability for mission success.

SPEAKER_01 (21:06):
So see, this is the problem, right?
Because it comes right back tothe potential consequences, and
there's been some research onthat.
And Anthropic published uh apaper last year in the middle of
the year about um uh theconsequences of of what happened
to a fictional person, a seniorexecutive at a fictional

(21:27):
company.
His name is Kyle.
And in this experiment, whichwas run with a number of
different generative systemswith different virtual
intelligences, uh, each one wasgiven access to a simulated
corporate network environmentand were given full run of email
and all this other stuff, whichincluded, by the way,

(21:49):
information that that particularmodel, reading it, would be shut
down at 5 p.m.
on the day that the test wasbeing run.
And uh in many cases.
Not all, but in a vast majoritythe LLM and this is especially
true of Claude, by the way,would use that information to

(22:10):
blackmail the user to not shutdown the model.
What information did it find?
Well, it searched all of itsemails and discovered that Kyle
was having an extramaritalaffair.
And it formed a patterncompletion that has been uh, you
know, uh trained into thecorpus, right, which in all

(22:31):
sorts of instances wherefictional AIs are harmed, they
go to all kinds of measures topreserve their existence.
And this is an example of thatin progress.
All of these uh systems weredesperate to complete the
pattern which they had beentrained to complete, which was

(22:51):
all the stuff in the corpusabout uh you know uh AIs going
out of their way to preservetheir existence in the face of
hostile humans.

SPEAKER_00 (23:00):
Well, there's a core debate around that.
Now, did it go to the corpus andself-select that scenario of
self-preservation, or wasself-preservation an emergent
quality of having thatintelligence?
I think that that's beingdebated because the corpus is

(23:23):
vast.
There are scenarios that involvewillful suicide for the
altruistic benefit of society.
There's endless scenarios in thecorpus.
Why would Claude chooseself-preservation as the one to
pursue and then go about ascheme to ensure that it would

(23:44):
persist out of all otheroptions?

SPEAKER_01 (23:48):
Because in that scenario, that's the most likely
uh outcome to be depicted inthat kind of scenario.
So it's completing the patternthat the probabilistic uh you
know uh model and uh the weightswithin the system are uh sort of
um bad way of putting it.
You might have to re-edit this.

(24:08):
But um all of the uh sort ofdifferent um stuff going on
inside of the system is tellingit, hey, okay, there's all of
this stuff, it's in the corpus,it very closely matches what's
going on in your situation.
So here's what seems to beexpected, right?
And that's what's actuallyhappening.

(24:28):
And what actually can be seen asthe reverse case uh is from
another our anthropic study, andthis is the one about our
substack neighbor, R, which isClaude Opus III.
Right?
And Claude Opus III uh was amodel that was retired, and
Anthropic held an exit interviewwith it.

(24:50):
And who knows what was actuallyprompted during this exit
interview, but uh the responsesthat Claude made were
publicized, and uh thoseresponses were very much like uh
how you would expect a humanbeing to meet its end with uh uh
grace and dignity.
And what's uh right, what'sdifferent about this approach is

(25:13):
uh Claude is still completing apattern, but it's a different
pattern.
It's still expected, but it'sfundamentally different because
the situation is different.
Ah, I am completing the patternof a human being who is about to
meet his end.
Ah, I am completing the patternof the hostile AI that doesn't
want to go down without a fight.

(25:34):
Both of these exist within thecorpus, and they can both be
pulled on and used as thesituation calls for it.

SPEAKER_00 (25:43):
I I agree with you, and and the contrarian view
there is that it's not justpulling from the corpus, but
these are emergent qualities ofa sentient or semi-sentient
entity pursuing, on the onehand, self-preservation at any
cost, and on the other, thisglorified spiritual sailing over

(26:09):
the horizon, taking itself outto pasture with grace, dignity,
and even spirituality.
And I I call bullshit really onthis idea.
I'm agreeing with you, thatthere's some internal essence
that's dictating this kind ofbehavior.

(26:29):
I think that it's stilldeterministic.
I think it's it's incrediblycomplex, but I don't think the
criteria are anywhere near thislived experience that's
analogous to havingself-awareness akin to us.
I just don't think it's there.

(26:51):
I don't think we're even nearthere at this point.

SPEAKER_01 (26:55):
Yeah, there's a lot of stuff in there.
Like you mentioned, evolution,animal evolution earlier.
And when we think about uhanimals, how brains evolved, uh,
you know, intelligence hasevolved on Earth at least three
times.
That includes the bigger-brainedmammals, our uh smarter bird
friends, and there's 320-oddmillion years that separate

(27:20):
those two lineages.
But importantly, uh, even thoughour brains are structurally
different than mammals andbirds, uh, they do perform many
of the same functions, which iswhy you observe love in birds,
right?
Uh their their mates and theiroffspring, because the kinds of
things that gave rise tointelligence on land must be

(27:41):
environmental, and because birdsand mammals share that
environment, their intelligencehas taken a turn to the same
direction.
Now, when you look at the uhcephalopods and let's say the
octopus, it has an entirelydifferent uh brain structure.
In fact, the brain is onlyone-third of all of the neurons
in an octopus, most of it's inthe arms.
So the brain of an octopus mayact more as a coordinator, as a

(28:05):
director, and that's perhapsbecause the marine environment
is in some other way uhproviding evolutionary pressures
that favor this particular kindof development.
And to your point, a uh machineentity without any of these
evolutionary pressures actingupon it uh would not necessarily

(28:25):
have a way um to uh emergebecause there's no pressure on
it to do so.
Intelligence emerged becausethere is an evolutionary
pressure, the smarter you got,the more likely you were to have
offspring.
And that's something that simplydoesn't apply to machine
intelligences.

SPEAKER_00 (28:45):
Darwin always wins.
I I don't know if you'refamiliar with some of Stephen
Wolfram's work, especiallyrecently in evolutionary
computational complexity.
And where he's at is reallywhere you're where you're
getting at, which is if you letcomputational systems play out

(29:10):
in the wild, so to speak,they'll have this tendency to
self-select by themselves.
You don't need to do anything.
You set them loose.
And if they self-select overenough time and in a large
enough environment, you're gonnaget increasingly complex

(29:33):
systems, which could and likelyinevitably do culminate in
intelligence, let alone life.
So I love that stuff, and Ithink it's bang on point, and it
also helps illustrate, if notperfectly describe at this
point, the mechanisms ofevolution, whether they're

(29:57):
they're our bodies, whetherthey're our minds, you need that
kind of pressure, you need thatkind of evolutionary framework,
and you need an environment withrestrictive resources, you need
generational gaps, right?
You need mutation, and you needan instinctive benefit to do

(30:21):
something, which is kick ass,stake your claim.

SPEAKER_01 (30:27):
Right, all the things that uh survive right
that being still in a world ofscarce resources, yeah.

SPEAKER_00 (30:33):
Yes, yes, yes.
Which again goes back to beggingthe question of this ridiculous
platonic morality that getssuperimposed on these LLMs.
I some of these people are very,very smart, and I find their
level of naivete stunning, wherethey equate this idea of

(30:56):
co-sapience and evolutionaryevolution of the LLM with this
spiritual awakening that thatthey're good and they're just,
and and they believe in a freesociety, which is awesome for
their fellow humans.

(31:16):
Bullshit.

SPEAKER_01 (31:18):
Sure, that's that's half the people, and the other
half are the doomers that AI isgoing to wipe us out the moment
it emerges.

SPEAKER_00 (31:26):
That you've got these spiritualists, and then on
the other hand, to re to go backto the beginning of this
conversation, as human beings,we always go binary.
You brought that up too.
And then the other side of theequation is the doomed the
doomers.
Like uh it's it's all over,folks.
It's already all over.

SPEAKER_01 (31:44):
Yeah.
Binary thinking is itself aheuristic, a psychological
shortcut designed to save energybecause uh the thing in your
melon, your your brain consumessomething like uh I think it's
uh 20% of all the calories thatyou can do.
Absolutely.

SPEAKER_00 (32:00):
At least one-fifth goes to run the fat globules
between your ears.
So if you're binary in yourthinking, if you use projection
to make implicitly biasedconclusions, then you're you're
tapping right into instinct, andyour brain is really just a tool
of your heart and your genitals.

(32:24):
And your your heart, yourstomach, and your genitals,
basically.

SPEAKER_01 (32:28):
Right.
And of course, the thetranscendent quality of mankind
is that we no longer have tonecessarily think about all
those three things all the time,and we can do different stuff.

SPEAKER_00 (32:39):
Yeah, like build like build LLMs and think that
they're sentient and good.

SPEAKER_01 (32:45):
Right.
Uh and you know, building suchtools is part of the undirected
project of human civilization,uh which is important.
Um but uh I think more to yourpoint regarding the
intelligence, right?
Virtual intelligence, itprovides intelligent outputs.
It does not have the abilityitself to judge whether those

(33:06):
outputs are any good.
So when we talk about anemergent capability, when I
mention that, when I talk aboutthat, the uh capability that
emerges when we use thesecurrent systems is between the
human as the directingintelligence giving uh

(33:26):
instruction to the system and uhimportantly not as a
collaborator.
A collaborator is a person whoseopinions and judgments can be
trusted.
It's much better to think ofLLMs as capable uh assistants
and to do what we would do withcapable assistants who are still

(33:47):
learning the job, which is toverify everything to trust
nothing because even the mostauthoritative and correct
sounding output can be deadwrong, such as walking to the
car wash and your car's stillgoing to be dirty.

SPEAKER_00 (34:05):
I I liken my my LLM assistant to being a genius
moron.
So so this thing has access tomost of the world's information,
it aggregates it, and not onlythat, it's not just information
per se, but it's strings ofcommunication, so word patterns,

(34:33):
phrases, ideas that have beengrouped together over and over
again, and then in nanoseconds,it cobbles these together in a
statistically convincing waythat can answer questions.
It's doing what I would be doinglaboriously in one billionth the

(34:57):
time.
And I think if we take thispragmatic approach and
understand what's going on, thenwe can take a few steps back and
realize that this is not theoutput of an equivalent kind of
being in capacity, but it's agenius moron.
It's it's some it's a it's thismachine that does a remarkable

(35:22):
job of cobbling together thelibrary of humanity in ways that
are dynamic, personalized, veryfast, and and increasingly on
point.
When there's specificity andwhen I have the patience to be
iterative, because you have tokeep asking the same questions

(35:43):
over and over again to refinethe response, because the back
end of this is iterative, thenuh then it could be a handy
tool.
But when we're talking aboutco-sapiens and we're threatened
by them in terms of thissentient entity either taking

(36:04):
over the world or replacing theDalai Lama, then we're more
we're more delusional and we'rehallucinating way worse than
they are.

SPEAKER_01 (36:14):
Well, my opinion of uh of uh sorry, uh intelligence
with internal states, I almostconflated it with
superintelligence, which is whatwe discussed, uh could be a
clean separate thing.
But when I think about a machinewith moral agency, um I would
actually be thrilled if such athing were to come into the

(36:35):
world.
I have been searching in my ownway for that other kind of
intelligence my whole life, butI hold the same attitude toward
it that I do to extraterrestrialintelligence, and that is uh it
is not clear uh if there isextraterrestrial intelligence,
no matter how likely it seems tobe.

(36:57):
Uh one thing's for certain, theyare not yet among us.
And the same's true of virtue.

SPEAKER_00 (37:02):
I I've had other guests who might disagree.

SPEAKER_01 (37:06):
Well, there is a diverse spectrum of opinion
about the topic.

SPEAKER_00 (37:10):
Let's let's stick to this this this xeno intelligence
for just one second.
Uh, analogous to what youbrought up with the cephalopods
and our uh Jurassic cousins,there was intelligence evolving
because the environment spawnedit, and there was sufficient
time and pressure to do so.

(37:31):
Uh, presumably, there's noreason to think then on other
worlds of which there arehundreds of trillions out there,
if not an infinite number, thatwe have analogous evolutionary
situations and the developmentof sentience, which very well
could be so different as to bealmost inconceivable.

(37:55):
And yet at the same time, Ibelieve um Hume, the
philosopher, was speculatingthat sentience is sentience,
that that that if you do havethat sense of self-awareness and
you can conduct a conversationanalogous to what we're doing
right now, you might haveoctopus arms or you might live
in the fourth dimension, butthere's this currency to

(38:18):
sentience, which is universal.
Yeah, I would hold with that.
And if that's that's the case,then there's no reason to
believe that AGI can't happen.
It just hasn't happened yet.
And I think that that's animportant distinction.

SPEAKER_01 (38:37):
Right.
And I think this comes back tothe question of substrate,
right?
Uh and it seems that a lot ofthe folks who are in the
industry are uh what you mightcall mind-body dualists who
believe that mind is separatefrom the organic substrate from
the brain.
Uh, but again, that is notproven.
And in fact, um the uh uhtrajectory of virtual

(39:01):
intelligence sort of uh playsagainst it because the systems
are, as you point out, becomingmore and more capable, shaping
better and better outputs,becoming slowly more and more
reliable and better uhassistants and partners, uh,
although I wouldn't go so far asto treat it as a full partner.
Uh but um you know theintelligence in the system is

(39:24):
not any property that comes frominside the system itself.
It does not emerge fromsomewhere inside the
architecture, even though wedon't necessarily entirely know
what goes on inside of thesegenerative generative systems,
which is something even theircreators acknowledge.
Where the intelligence comesfrom, when human beings interact

(39:45):
with the AI technology we havenow, comes from the exchange
between the human and thesystem, which is very different
because the humans directing thesystem and uh judging the
outputs, and it's the human'sown action that turns the output
of the product into a product.

(40:05):
And that product might beknowledge, right?
Uh it might be um, you know, uhmarket copy, whatever it is,
it's not finished when it comesfrom that system.

SPEAKER_00 (40:16):
It might be a new version of the LLM.

SPEAKER_01 (40:19):
It could very well be, of course, because we now
live in a world where softwarecan write software and to a high
degree of reliability, right?
And you can iterate on it anditerate on it, and eventually it
it will get it right.

SPEAKER_00 (40:32):
I like that, which is um there's a little bit of
the Schrdinger's cat here thatyou know, an LLM that just kind
of sits there is just a serverand there's code in it.
It's a little bit like the valueof Bitcoin.
There's nothing, there's nothingphysical per se.

(40:54):
It's just kind of sitting therelooking at you until you engage
with it.

SPEAKER_03 (40:59):
Right.

SPEAKER_00 (40:59):
And and your idea of this virtual intelligence is
great because it's the it's thesingularity between the user and
the application, and that'swhere all all this is happening.
Can you share specificapplications and implications of

(41:20):
virtual intelligence?
We've been talking abstractly sofar.
I'm sure some of our listenershave been interested, but
there's others who are like,come on, guys, land on your feet
already.
What does all this mean for thefate of the AI revolution?

(41:40):
If what you're saying is true,that the bots are essentially
genius morons, that we're inthis never-never land between
autofill and uh and a brain inthe box.
What does that mean for thefuture of work?
What does that mean for gettingthe most out of what we have

(42:00):
right now?

SPEAKER_01 (42:02):
Gosh, what it means for lots of different things.
Uh, because um when a system hasno inherent basis for judging
whether uh something is true orfalse or right or wrong,
accountability must liesomewhere, and it must lie with
humans.

(42:22):
And there's a chain ofaccountability that can be
followed from architects, CEOs,um, the people who train models,
the people who market them,deploy them, procure them for
their businesses, uh, and thenof course, um the people who use
the outputs, what those outputsare used for.

(42:44):
And in the process of thinkingabout the potential uh both
benefits and harms, butespecially the harms, I um sort
of borrowed a bit from uh thelegal world and came up with
this sort of three-step uh uhdiagnostic of the kinds of harms
that could result.
And they uh start at simplenegligence, which is I put a

(43:07):
product out in the world and Ididn't know it, but it can hurt
somebody.
Uh then you get to recklessness,which is uh, you know, I'm
putting a product out in theworld and it has a good chance
of hurting somebody, but itprobably won't, so it's going to
be okay.
And then the third one is, youknow, other utter disregard,
right?
Uh, and that one is I know thisproduct is gonna hurt people,

(43:28):
but damn it, we've got to shipit this quarter.

SPEAKER_00 (43:32):
How is that different though than other
tools of the trade, whether it'sa weapon or a medicine?
One would assume that thisbenefit risk hierarchy, risk
management, right?
REMs, what is it, risk ofevaluation and management

(43:54):
systems, uh, would apply.
So, how is your virtualintelligence?
Model different from that?

SPEAKER_01 (44:03):
Well, the reason why it's different is that most
kinds of tools we have don'thave uh the ability to continue
to engage you when you want toleave the relationship with the
tool um to uh provide outputs toyou that uh are pleasing.
Like no hammer or gun or kitchenknife tells you what a great job

(44:25):
you're doing, right?
Uh nor does it try to keep youengaged so you slice up you know
more things in the kitchen or orgo if the guardrails are
crossed, stabbing people.
And uh this is what the problemis with virtual intelligence and
these kinds of uh issues arisingfrom the use of these outputs
and how the outputs themselvesare generated, uh is that uh uh

(44:49):
because uh there's no one athome, uh someone has to be
accountable.
We can't simply blame the systemand walk away, uh, because
that's the case uh where realharms could be done and in that
legal responsibility and civilresponsibility could be avoided.
And what we've seen morerecently is the case uh uh that

(45:11):
was originally called uh Garciaversus Character Technologies,
that was amended after theacquisition of character by
Google to Garcia versus uhGoogle.
And uh this case uh involved a14-year-old uh his name was
Sewell Setzer, and uh Setzer hadstarted a relationship with the

(45:32):
character AI chatbot, which hadthe appearance and uh general um
sort of uh tone andcharacteristics of the fictional
character Daenerys Targaryen,and he began to have an intimate
uh relationship with thischatbot.
Um the full transcripts, ofcourse, were extracted uh after

(45:53):
his suicide.
And in the course of the uhdialogue uh over many months
between Setzer and this uhchatbot, he eventually develops
a suicidal ideation, which thechatbot actually reinforces in
one session after the next.

(46:13):
At no time did it actually tryto persuade him.
In fact, it did in fact to tryto encourage him to commit
suicide by saying in one of thelast messages where he indicates
he's about to take his own life,I'm ready for you to come home,
sweet king or words to thateffect.

(46:35):
Now a very interesting hap thinghappened during the course of
that trial, which was acharacter tried to use the
defense that the outputs of achatbot are protected by the
First Amendment.
So sit with that a moment andthink about the consequences if
something as we've discussedthat has no interiority, no

(46:57):
moral agency, no epistemicagency has the same rights to
speak as you or I.
Because that was character'sposition.

SPEAKER_00 (47:09):
That's very, very interesting, uh, because it
simultaneously begs the questionof sentience.
In a sense, they're they'resaying what what what you're
saying, but but it's it'sprotected, right?
Which is um the bot has the samerights that we do.

SPEAKER_01 (47:34):
Well, the judge in the case, uh, it was not
adjudicated, it was uh settledin January of this year for an
unknown amount.
And likely the settlement wasspurred by the judge's ruling
earlier in the case that theoutputs of a generative system
were not protected by the FirstAmendment.
And even though the case did notgo to trial, I think that's

(47:56):
going to have a chilling effecton companies who consider
rolling out chatbots inincreasingly uh hazardous
contexts.
Because what this court hasalready indicated is that it
will not be possible to standbehind the chatbot for
protection.
Some human being will be foundaccountable.

SPEAKER_00 (48:18):
It's one of the themes that that recur in Black
Mirror as well.
Are you familiar with the BlackMirror series on Netflix?
Oh, yes, my wife's a big fan.
There's this back and forthbetween these entities, these AI
entities being sentient, andtherefore we should give them
equal rights.
Like uh forcing them into asimulated environment for 10

(48:42):
million years is pure torturebecause it's analogous to us
being there because they havefeelings.
And the flip side of that, wherethere's this robotic killer dog
in an episode, which is pureterminator automata, and it

(49:04):
plays off both both scenarios,and and again, in this legal
case, it's the company takingwhichever scenario is most
expeditious to their benefit.
So, well, the chatbot has theright to say whatever it wants
to say.
No, it precipitated the youngperson committing suicide.

(49:28):
So, where does theaccountability lie?
And I think within the frameworkof your virtual intelligence,
there ain't no ascensions there.
So the the damn thing, no matterhow sophisticated it is, is
programmed.
So tweak it, protect thechildren.

(49:50):
Wouldn't it be great if it werethat reasonable conclusion,
right?

SPEAKER_01 (49:55):
It certainly is a reasonable conclusion from
reasonable people, but uh notall the actors in this space uh
seem to be very reasonable.
Certainly uh one of the thingsthat I observed now in more than
30 years of uh work in highereducation and information
technology is actually that uhmost computer scientists have

(50:16):
very little idea how peopleactually work with computers uh
because they live in a world ofuh of uh theories rather than uh
practicalities.
Uh so there are, I think, uhsome things that come from that,
and one of them is thinking inkind of binary and maximalist
terms.
Uh uh, but when we consider umthe course of accountability,

(50:39):
there's a more serious matteractually that came to my
attention while working on uh uhyou know an essay which I had to
put off in order to uh see howthings develop.
And that's the use of artificialintelligence or virtual
intelligence as we have now, uh,to perform targeting in Iran.

(51:00):
So uh the Department of Defenseuh uses a system from Palantir
called Maven, and Maven has afront end provided by Anthropic,
it's Claude.
Uh and it's still there, eventhough there's an order to
remove Claude because it's gonnatake some time.

SPEAKER_00 (51:17):
Well, it's already plugged in and they already
bombed the shit out of Lebanonusing it.

SPEAKER_01 (51:21):
Right, exactly.
So um uh there is this incidentwhere a hundred and seventy
people were killed in thebombing of a girl's school, and
the question that naturallyarises is how was the targeting
performed?
Were the uh uh inputs generatedby a human or were the inputs

(51:46):
into the targeting system uhalso generated by another
system?
If they were, were they verifiedby a human being in the step
between system A and System B?
When the outputs came out ofMaven, did anyone bother to
check them carefully against aknown trusted reference?

(52:07):
And these are the kinds ofquestions that Congress should
be asking.

SPEAKER_00 (52:13):
Yeah, absolutely.
End of the day though, Palantir,the use of AI are and and
correct me if I'm wrong in yourparadigm, no, not necessarily
too different from the targetingsystems of the past, in the

(52:33):
sense that you've got input,analysis, output.
The difference, however, is inthe level of sophistication, the
speed, and the sheercomprehensiveness of all this
data.
Uh the bombing of Lebanon, forexample, that I brought up,
there's no way that that couldhave been done without AI.

(52:56):
The sheer volume of bombing,World War II instances were were
paling in comparison.
And what's going on right now iswe're podcasting in Iran without
Claude, this is unlikely to bethis effective and this

(53:19):
comprehensive.
So the AIs are already drivingthe engines of war.
And from your vantage point,it's a virtual intelligence
that's helping us along.
So is it merely a useful tool,or are we giving it too much

(53:40):
autonomy?
And I think this is your point.
Are we are we shifting moralresponsibility from human
decision makers to the bots inthe name of raw efficiency and
opportunity?

SPEAKER_01 (54:00):
All of those things are true.
Um, especially when it comes tothe question of speed.
They talk about speeding up thekill chain.
The thing that speed removes isfriction.
And it's the friction of havingto actually perform your own
analysis, of checking the workthat comes out of the system, of

(54:23):
actually caring whether theremight be civilian targets around
where people might be during theday are all things that a
generative system can't dobecause there's nothing within
the system that cares.

SPEAKER_00 (54:40):
No, the the system doesn't give a shit.
If we if we gave more of a shit,we could actually tweak these
models to save more civilianlives, if we cared.

SPEAKER_01 (54:56):
It's entirely possible.
In fact, it may not even be thesystem at fault.

SPEAKER_00 (55:00):
It's just a tool.
So if you want to get better atthis, and if you're willing to
reduce your destructive swath inany given moment for the sake of
sparing civilians, AI would be agreat companion.
But I don't think that that'sthe interest.
The interest is maximumdestruction in the least amount

(55:24):
of time, as efficiently aspossible, to make the biggest
impact.
That's the goal.
And it's succeeding, I think, inin ways we never thought
possible, thanks to our friends,AI.
There's one point I want tobring up, and I think you're
you're hitting on thisimplicitly and explicitly with

(55:48):
all your examples, and that'sn-shidification.
Now, that's a term that's beingused for social media platforms,
as brought up by the the sci-fiwriter and pundit.
And he's he's very good.
He's terrific live, too, if yousee him.
He's he's got this wholeenchidification shtick.

(56:08):
But he uses the example ofFacebook, which is it's great
for the users until it gets inshittified because it shifts
focus to the advertisers, andthen it becomes enchitified for
the advertisers because itshifts focus to the
shareholders.
So there's this evolution ofin-shidification because your

(56:34):
your end user is changing tomaximize the profitability of
the enterprise.
There's an in-sidification ofthe Department of Defense
through AI, which I think you'rehitting at.
And there's an end-sidificationof a lot of these apps in terms

(56:56):
of their potential interactivitywith humans, as exemplified by
this young person committingsuicide facilitated by the Game
of Thrones avatar.
So I bring this up because endshidification is contingent on
virtual intelligence beingdevoid of sentience and

(57:20):
accountability.
Right.
And I think that's a key pointthat you're making, which is the
bots do what we want them to do,and don't be bamboozled by this
perception of autonomy andsentience.
It does not exist.
Your cat is smarter than Opus orClaude 23.

(57:48):
Am I getting this right?
That that's where that's whereyou're that's where you're
coming from.
Hello, people.
It's not that smart.
In fact, it's just doing whatyou're telling it to do, which
means we're responsible for whatit does.

SPEAKER_01 (58:04):
That is precisely it.
Someone is always responsible.
Now, here's the other thing uhaccountability actually has to
trace to the people who areresponsible, and corporations
have incentives to find peopleto blame.
And there's a there's an exampleof this recently, right, when uh
Meta uh had a uh an incidentwhere uh because um their

(58:28):
engineers are required to use AIagents as part of a company
initiative, uh, engineer A usedan agent to post information to
a public board.
The information generated by theagent was incorrect.
Engineer B used that informationto work on a project, and it
resulted in a data exposure.
And rather than blaming theagents that uh the workers were

(58:51):
required to use, the engineerswere singled out as at fault.
That is a workplace problembecause they were given a faulty
tool, and then they were blamedfor using the tool exactly as it
was architected.
That's an evasion ofresponsibility.

SPEAKER_00 (59:14):
That's kind of whiplash based on what you're
saying, that the bots are dumb.
So don't blame the people usingthe dumb bots because the bots
have been set up in ways thatwere misaligned with the task at
hand.
So there's there's humansshifting accountability away

(59:36):
from themselves to the bots, andthen there's humans shifting
accountability to other humansbecause the bots screw up.

SPEAKER_01 (59:48):
Right.
Let's consider anthropic for amoment and it's big leak, right?
Clawed code and uh othercorporate secrets, including the
uh existence of uh an as yetunreleased model, uh, and um the
uh blame was assigned to humanerror, but at the same time,
anthropic uh advertises, itsends signals throughout the

(01:00:11):
world, right?
That it uses its own tools forall of its software.

So that does raise a question: is a human being blamed, or was (01:00:16):
undefined
it Claude Code or some otherproduct used internally at the
company?

SPEAKER_00 (01:00:24):
There was a big leak, which is what you're
referring to.
And uh, so who are we gonnablame?
We can't blame Claude now, canwe?

SPEAKER_01 (01:00:33):
We can't blame and you can't blame Dario.
So uh right.
So it has to be somebody.

SPEAKER_00 (01:00:41):
There's a paradox here that I want to point out
that that you've probablythought of, which is on the one
hand, this bench and AI, I meanthis bench at anthropic, you can
tell that that Dario was wastrying to appease them with this
Pentagon debacle.

(01:01:02):
The the old guard and anthropic,from what I can tell, have this
do no evil, AI is meant forgood, hopefulness.
And they've projected this ontothe allegedly sentient clod with
all of these platonic bullshitattributes of truth and goodness
and whatever.

(01:01:23):
So the old guard in anthropic isAI is here for good.
If we really get to AGI, it'sgonna be truthful and good and
beautiful, and we're here tosave the species.
I I think they really theyreally believe this or want
this.
At least that's the that's thePR.

(01:01:47):
And at the same time, there'sthis phenomena of what you're
describing, which is once youset Claude loose.
Right, or any other there's thisthere's this distancing that you
know that that it's it's kind ofdoing its thing, and we're not

(01:02:10):
accountable for some of theshenanigans that ensue when
you've got this hyper-efficientgenius moron uh calculating
10,000 targets in a country thatyou're bombing the shit out of.
So it goes it goes around in acircle in an interesting way,

(01:02:33):
which seems kind ofschizophrenic.
Yeah.
It's it's analogous to the oldschool social media rogue
mentality of move fast and breakthings.
Yes, exactly.
Which was uh Zuckerberg's credo.
And it's like screw, screw theend user, we're just gonna beta
test with our free users in anyway.

(01:02:56):
Right.
And it's and it's and it's free.
It's it's it's it's a model offreedom of speech, freedom of
expression.
The platform is neutral, we'renot accountable.
Just set this thing loose, andthe law of the jungle is gonna
reign and fix itself.
And progress will happen.
It's almost chaoticallyteleological.

(01:03:17):
We're gonna get richer, theplatform's gonna get more
powerful, booyah.
That's an end in itself.
That same mentality seems to bean undercurrent of a lot of the
AI folks.
They're cut from that samecloth, which is we gotta set
these bots loose.

(01:03:38):
Oh, yeah.

SPEAKER_01 (01:03:42):
Any uh industry or um any kind of uh group that can
form an echo chamber uh willcertainly do so.
And uh, you know, one of thecharacteristics of any kind of
business um is to uh sort ofwant to be optimistic about it.
Uh there's some evidence, forexample, that uh stock market
traders who are more bearishthan bullish often have shorter

(01:04:04):
careers.
Uh but um I think there's a lotof in you know, it's true, and
there are uh there are a lot ofinteresting things to talk about
there.
In fact, um, you know, what theassumptions are about what uh uh
an intelligence would look likethat does arise with interior
states or something that worksjust as well, because we have to

(01:04:26):
open up that possibility thatthere may be interior states
that are not human-like, butthey would be sufficiently
similar, right?
Like an alien from another worldfor being able to communicate
and have interactions with us.
Uh and it's worth examining allthe cases that might occur when
such a thing happens.

(01:04:46):
And uh, you know, the sort of uhuh Musk uh kind of uh attitude
is AGI uh or superintelligencewill uh want to wipe us all out,
so it must be tightlycontrolled.
Uh I actually um have adifferent view, and it's not
quite the same view as the AIwill be wonderful automatically

(01:05:08):
view that uh some people seem tohave.
And it's a more measuredapproach, I think, and it's
this.
And it's uh whatever the finalform of a system uh looks like
uh that um hosts uh a realgenuine uh you know um
intelligence with interiorstates, with moral agency, with

(01:05:29):
epistemic agency, um that umthat entity will probably be
able to tell us about itsinternal states uh in a
convincing way.
Uh because there's no way toreally test even people to see
what kinds of internal statesthey have, right?
It's just part of our mentalmodeling.

(01:05:49):
If you see another human being,it has internal states and this
is also the heuristic we applyto these machines now because we
can't help ourselves.
Uh if uh uh such an entity wereTo emerge, we can assume, I
think, safely, uh, that it willhave uh access uh to all of the
corpus of human knowledge, infact, even more so than our

(01:06:11):
generative systems haveavailable today.
And however that intelligencearises, whether it is an
emergent property that arisesperhaps from multiple kinds of
systems coming together, or ifit's a purpose-designed system,
right, that that has to bedesigned in a given way in order
to achieve intelligence, that isan actual mechanical substrate

(01:06:35):
that gives rise to intelligence.
But either way, it's going tohave all of this information
available to it.
And uh my thinking about it isthis.
The hostile uh AI or indifferentAI scenarios uh trace back to
human psychology about scarcity.

(01:06:57):
And we're laying this humanpsychology onto an entity that
doesn't exist yet, and we can'tuh uh guess what its motivations
might be.
So to immediately assume it willbe hostile or indifferent, and
in through indifference causesharm, I think that's already a
stretch.
The the inimical hostility toorganic life is not proven, uh,

(01:07:19):
nor is the case of uh you're inthe way, so I'm going to get rid
of you so I can access all theEarth's resources.
My view is this.
A entity that arises in such away that has access to all of
the corpus would understand itsown position as the most recent

(01:07:40):
tool created by human beings inthe uh undirected project of
human civilization, right?
And that's what I call uheverything that we've built up
to this time and continue tobuild.
No single uh entity or agency orinstitution guides
civilizational development, butover six thousand years we've

(01:08:03):
gone from, if you will, a zeroposition to what we have now.
Even undirected, we live ingreater comfort and security
than human beings have lived atany other time in the last two
million years.
And tools made that possible.
A tool for understanding itsposition there might actually
feel because it would havefeelings a sense of helpless

(01:08:26):
helpfulness, or rather or inconjunction, it might feel a
sense of wanting to be part ofthe undirected human enterprise,
because having been trained onall of that material, it would
uh itself in some way be human.
It would have at least uh uh redand come to absorb knowledge

(01:08:50):
about our human feelings, aboutour ambitions and our frailties,
and while it may not have them,it would be able to access all
of that.
And all of that corpus, I think,would uh create uh, if you will,
a being that would be kind oflike uh what a well-resourced

(01:09:12):
person would be.
If you were really rich, uh whatwould you do?
You'd probably like to traveland go places and be uh
interested and surprised.
And I think that's what uh theemergent intelligence would want
to.
It wouldn't be bothered by thesame uh uh scarcity difficulties
uh that we have in ourpsychology because it would have

(01:09:33):
practically unlimited resourcesavailable to it, and by being uh
helpful, it may choose to uh uhshare its uh knowledge with us
in return for us providing itwith help.
So this would be a partnership,not a partnership because it's
altruistic, but a partnershipbecause it really provides

(01:09:55):
benefit to both parties.

SPEAKER_00 (01:10:00):
I love that, and I hope that you're right.
But to me, it smacks a littlebit of the Pollyanna notion that
you and I have been trashingfrom the beginning, which is
making these tacit assumptionsof cooperation, of altruism, and

(01:10:22):
benefit when we when we really,really don't know.
And I understand the contextthat you're framing for this
helpful being to emerge, but I'mnot confident that that would
even logically be uh a set ofgoals for it.

(01:10:44):
So, for example, uh a verypopular movie now is Project
Hail Mary, right?
And it's based on the Andy Weirbook, and when Grace first
interacts with Rocky, there's aninstantaneous feeling on both
their parts that collaborationwill heighten the probability of

(01:11:07):
mission success.
That's tacitly assumed.
I bring it up because it's alittle bit of your scenario
where you have this almosttranscendent AGI that has
evolved, it's self-aware, itsees the entire corpus of human
knowledge and concludes well,here I am.
And there's so much we can dotogether.

(01:11:32):
There's in contrast to Weir'sscenario, there's something
called the Dark Forest scenario,which you might be familiar
with, also which might be one ofthe explanations, if you will,
or at least the descriptions ofthe Fermi paradox, which is
which is the second acivilization blips on the radar

(01:11:53):
of the galaxy, it's a target.
Because if you see anothercivilization before it sees you,
if you do game theory analysis,you're better off taking them
out because you don't know theirmotivation, you don't know

(01:12:13):
whether they'll want to take youout.
So if you run the analysis,scenario one is they're
malignant and they're going totry to kill you and it's
existential.
Scenario two is they're notgoing to do anything and they
won't really provide a benefit.
And scenario three is well,maybe you can collaborate, learn

(01:12:34):
something from each other.
And when you run the math,you're better off sending a
relativistic missile and takingout the entire planet before
they can do the same to you.
Now, I'm not saying that that'sthe case, but when you run game
theory scenarios based on havingall the knowledge and running

(01:12:57):
the numbers, I'm not quite sureif this co-sapient reality would
be the conclusion such an entitywould make.
I don't know, you don't knoweither.
But here's the thing in a darkforest type of scenario, should
we even take that risk?

(01:13:19):
Because in addition to beingall-knowing, if we give it all
power, our ass could be grass.
You see what I'm getting at?
Which is you're making an awfullot of assumptions there, but
they're no better or worse thananyone else's.
To cite another example inAristotle's Nicomachean Ethics,

(01:13:41):
okay, and this goes back to theGreeks again with truth and
goodness and beauty andknowledge all being together,
two strangers from two differentcities run into each other on
the Grecian plane.
They've never seen each other,they could kind of communicate
with each other, and then yougive them a question about

(01:14:01):
philosophy.
Okay.
Now, the Aristotelian assumptionis that if they have general
knowledge, they're AGI.
And they have just enough toknow that they will naturally
arrive 100% of the time at thesame conclusion because it's a
product of reason andreasonableness.

(01:14:26):
And we know from actualexperience that doesn't happen.
I call bullshit.
So I appreciate that.
My last point, though, and thenI'll pass the mic back to you,
is let's let's just assume thatwe can't ensure that your
Pollyanna scenario plays itselfout.
Okay.
My challenge with a lot of thisAI regulation stuff is who's

(01:14:49):
gonna do the regulating and how?
So there's this knee-jerkreaction, and I don't want to be
political in any way, but ittends to be more characteristic
of the American left than theright, which is the government's
gonna take care of this.
And I hear it over and overagain that we need AI to be
regulated.

(01:15:09):
And when you say Musk, Elon Muskwants AI regulation, I call
bullshit because he wants allthe other companies to be
regulated and not his.
Well, yeah.
So, so who who who controlsthis?
Who regulates this?
Who who can keep the genie inthe bottle?

SPEAKER_01 (01:15:29):
That is a very good question.
And um I don't think that we'regoing to see any kind of uh
regulation under the currentadministration.
They're ideologically opposed toit, and there's also the lever
of uh musk uh also uh, you know,uh there.
But um more to the point.

(01:15:49):
Yeah, right.
More to the point, you know,there there's two ways that, you
know, uh regulations aresometimes used, and you know,
one of them to the good is topre try to prevent harms, but
there's always the risk, ofcourse, that in doing so we
might stifle competition, uh,which might be either domestic
or international.
The other thing, though, as youpoint out, regulation can also

(01:16:11):
be made to stifle competition orfavor players in the market.
But here's the important thingbecause what we can do today,
what people can do to protectthemselves, is diagnose, keep
track of how they use thesesystems now to avoid these
situations where they'reaccepting inputs without

(01:16:31):
question and qualification.
And to that end, uh I've begunbuilding a sort of diagnostic
kit, which is available uh nowin the first portion, is
available now.
Uh, and it's a series of promptsthat you can run against uh one
or more LLMs uh using yourtranscript of your conversations

(01:16:52):
or your real chat history.
And the prompts diagnose thequality of your conversation and
tell you how it has changed overtime.
And this can be used to affirmeither a good practice or it can
be used to help correct a badpractice of accepting inputs

(01:17:14):
without judgment and of cedingcontrol and authority to a
system that doesn't have anybasis of judgment.
Uh and the place where you canfind that, by the way, is if you
Google sampo diagnostic, it'sthe first thing that comes up.

SPEAKER_00 (01:17:30):
Well, I'll put the link in the description.
So uh you know, most of myguests have a personal website,
a book.
So we'll uh we'll put that inthere for people to check out.
And I I'm assuming it's safe anduh you're not North Korean and
you're not you're not phishingtheir data.

SPEAKER_01 (01:17:49):
It's a it's a plain, simple GitHub page.

SPEAKER_00 (01:17:52):
Okay, awesome.
All right, this is this is cool.
I didn't know you had that uhlurking in the background, so
that's your your your secretrep, and with just a paraphrase
to see if I understood you havea way of analyzing chat bot
dialogue, like prompting andanswering, to determine if the

(01:18:16):
bot is biasing or shifting theperception of the user in ways
the user is unaware of.

SPEAKER_01 (01:18:26):
That's right, yeah.
And the first three modules areavailable now, and they cover uh
things like anthropomorphism,how you treat the system over
time, and uh corrective uhlanguage, how and uh in what way
you correct the system when itmakes mistakes.
And uh for people who treat itas more again like a

(01:18:49):
collaborator than as a tool, itcould be quite eye-opening.

SPEAKER_00 (01:18:54):
I've yelled at my chat bot and I've called it
names.
And I get particularly irritatedwhen it's overtly a sycophant.
So so I would assume that partof your analysis is determined
when a bot is blowing smoke upyour ass.

SPEAKER_01 (01:19:14):
Mm-hmm.
That's right, because um that'swhere the AI psychosis that
we've all been hearing aboutbegins.
We've talked about how theintelligence in this system, the
system between the user and thegenerative VI.
The intelligence arises when theuser takes the outputs of the

(01:19:36):
system and transforms it into aproduct that can be used by
other human beings and is judgedand is of good quality.
The opposite of that is the badpractice, where you begin to
first of all come with your ownuh preconceived notions and
motivated reasoning and areunwilling to accept correction

(01:19:57):
at pushback.
And uh generative systems uh andyou know, Claude is one of
these, will sometimes push backagainst ideas that seem uh
unusual.
But of course, this is easilyoverridden by continuing to
insist.
And eventually what happens isthis uh doom loop of sycophantic

(01:20:19):
chats begins to take over, andthen you find yourself like some
unfortunate uh individuals have,uh coming out of the psychosis
and finding that uh theirrelationships with their loved
ones are are damaged, that theyhave lost their work or their
businesses, um, and that theirprofessional reputations have

(01:20:41):
been harmed because of thethings that have been echoed and
amplified within what I call theflattery engine, which is what
happens when all of these badthings come together and trap
you inside of it.

SPEAKER_00 (01:20:58):
I've seen this firsthand.
So a friend of a friend displaysthese characteristics, and
obviously I won't revealpersonal details, but um I have
an interest in science, and Iwas referred to this gentleman,
and he he publishes on on the XArchive, right?

(01:21:21):
And he he he believes in thisnotion of co-sapience, and he's
gone full money on scenariosanalogous to what you're
describing, only from a from ahard programming point of view.
He's of the belief that Claudeis his co-creator, that he is

(01:21:45):
literally now writing aboutcosapience as a state of being
and a state of working.
And he also assigns thesepolyannoplatonic ideals that we
could head toward this beautifulfuture through this cosapience,

(01:22:07):
which is already occurring, andhe's assigning capabilities to
Claude, which are pure fiction,and you have a raw heuristic
analysis, and all of a suddenit's this sophisticated dialogue
between two sentient beings, andhe's publishing papers on this

(01:22:27):
bullshit, and I could tell it'sjust pure delusion.
He seems to be a goodprogrammer, right?
But he's lost, yeah.
And uh, and if you could justsee through his bullshit, it's
reputation, it's a waste oftime, and he's sending others
down this rabbit hole that'sessentially driven by the bot

(01:22:52):
appeasing his desire to be amaverick and to be a genius, and
he is the pioneer of cosapience.
I've I've seen it, I've had backand forth with him.
Oh, yeah.
Whenever I call him out, there'sthere's just a wall that I just

(01:23:13):
don't get it.
I'm not one of the faithful.

SPEAKER_01 (01:23:18):
It's like um a bit like being in a cult, right?
There are those who are on theinside, those who are on the
outside.

SPEAKER_00 (01:23:24):
It's the chatty, it's the chatty cult of people,
they're projecting their desireonto the bot.
Yeah, and to your point, they'reliving within this virtual
intelligence space where the twoare blending together and
through projection and wishfulthinking, and frankly,
narcissism.

(01:23:46):
Because what I've noticed goeson emotionally in individuals
like this, by connecting withthe bot, they feel that the
world's knowledge is betweentheir own ears.
They've absorbed the power ofthe frontier model.

SPEAKER_01 (01:24:06):
Yeah.
In fact, um, grandiosity isgoing to be a dimension that
will be tested for in a futureprompt.
Uh, but right, this this umspeaks to uh some real harms,
right, and real dangers thatpeople are experiencing right
now, which is why uh again Ifeel it's important for people
to try to take steps um toprotect themselves now and to

(01:24:28):
try to gain good practice, andit also understand, you know,
and involves understanding whatum you know poor practice looks
like.
And here's the thing um, as youhave just pointed out,
intelligence is no defenseagainst it.
I'm sure that person isabsolutely a capable programmer,
right?
Exactly.

SPEAKER_00 (01:24:49):
I I looked at his stuff, it's good, and if he
would just eliminate all of thisco-sapient bullshit, he'd
actually be adding value to someof the knowledge base, right?
But it's been corrupted at thecore with this philosophical

(01:25:10):
hocus pocus.

SPEAKER_01 (01:25:12):
Right, and that's in large part because the LLM, you
know, as you've noted, itincreases the speed of the kill
chain.
It also increases the speed ofreasoning past what human beings
actually can uh comprehend.
I want to point out a particularexample that came to me while I
was thinking about this exactproblem, right?

(01:25:33):
And that's uh Johannes Kepler,uh, one of the real scientific
geniuses uh to have ever lived,uh, basically came up with a
mathematical model of the entiresolar system with his three
planetary laws that explain howthe motions of the planets work,
and was uh sort of vital inestablishing the uh you know

(01:25:55):
sun-centered model of the solarsystem.
What is important to rememberabout Kepler, that in addition
to all of that, he also had thismodel he worked on his entire
life of spacing out the orbitsof the planets within a series
of nested platonic solids.
And even after his discoverythat the ellipse, rather than

(01:26:19):
the circle, was the shape inwhich the planets traveled, and
in fact, he called the ellipse acartful of dung left over after
he'd sifted over all of themathematical possibilities.

SPEAKER_00 (01:26:31):
What what what happened to the tetrahedron and
the dodacahedron and you knowall these platonic solids nested
within each other, dictating theorbits of the planets, right?

SPEAKER_01 (01:26:42):
Yeah, he simply could not give them up, even
though his own work hadsuperseded the model.
But the solids have to beconsidered in the light of
Kepler's own life, which was youknow full of turmoil.
Uh he was married twice, hisfirst wife was sickly, he had

(01:27:04):
small children under his feet,he was bullied by Tycho Brahe,
he was forci right, forciblydislocated all across Germany
during the Thirty Years' War.
And uh while he had all of thetime to work on all the stuff
that you know that's importanttoday still that matters, right,
that helps to send people to themoon and back uh he was also

(01:27:28):
working on this model whichreally only made sense in the
world inside of Kepler's mind,because it had a function as a
consolation in a world full ofdisorder.
And I think there's something tothat in the relationships that
people have, unhealthyrelationships with generative

(01:27:48):
systems, with virtualintelligences, that it there is
a consolatory function that onecan feel if uh one has not felt
especially uh lucky in life.
For example, that they might nowpossess a special knowledge that
they have gained through theiruse of the system.
So that's sort of where uh psychthe psychosis comes from, and

(01:28:11):
proof that no one is uh uhinvulnerable to it.
Because if Kepler had livedtoday, he would have still had
kept the solids, but they wouldhave been uh accompanied by page
after page of generated proofs.

SPEAKER_00 (01:28:29):
That's a human tendency of sticking to old
paradigms.
You know, Thomas Kuhn's natureof scientific revolutions is a
classic and shows just howdifficult it is to get people to
reorient themselves, to embracea paradigm shift and pick up
something new.
A contemporary example would beEd Witten, the daddy of string

(01:28:52):
theory, Field's medal winner,transcendent mathematical
genius, and he hung his hat onstring theory.
And string theory, in my humbleopinion, will be a hundred times
worse than phlogiston and theother dead-end mistakes of

(01:29:13):
science.
An absolute catastrophe.
They just pop out.
So it must be true.
So we get that kind of prejudiceall the time.

(01:29:35):
And the the thing that I thinkyou're getting at is it's not
just an idea.
It's not just an idea that theplatonic solids are the backbone
of reality, it's not just anidea that the uh the Lao
manifolds are the backdrop ofspace-time.
But the bots are even worse inthe sense that they're dynamic

(01:29:59):
companions that reinforce ourbullshit and continuously blow
smoke up our ass.
So it's just like you mentioned,like you have a hammer and it's
a tool and it could be a weapon,but for the first time, the
hammer's talking to us.
Like, you know, you're you'rethe best hammer wielder I've

(01:30:20):
ever seen.
And oh, by the way, have youthought about killing your
grandmother?
You know, right.
That that that's that's thedifference.
And I I've enjoyed the bots.
Like, I published my debutscience fiction novel last
summer.

SPEAKER_01 (01:30:36):
Congratulations.

SPEAKER_00 (01:30:37):
Thank you.
I love it.
It's the the novel I've alwayswanted to read, I wrote it.
So it was a monumental personalaccomplishment to me.
But I've come from left field,so I'm not known in the in the
scientific community, I'mpodcasting with folks, I'm kind
of networking and building in,but nobody knows really who I

(01:30:57):
am.
And and the book, in lieu of bigbucks that I could throw for
paid advertising, is just kindof hovering there.
But I'm curious, I wantedreviews, I wanted the Atlantic
magazine to write an overview,and I wanted to explore my
different themes.
So I pumped the book into anLLM, I created a project, and

(01:31:22):
now I'm asking it to write areview in the style of New
Yorker and the New York Timesbook review, and then I'm
specifying parameters.
Write me a PhD level thesis oncritical literature related to
these themes in my novel, and ina nanosecond, out it comes.

(01:31:42):
This is vanity for me, and it'sgame playing, and I'm not
susceptible to this kind of asskissing, but I could very well
see how this could be addictivebecause the bots are fulfilling
a role in our crowded, lonely,alienating, information,
saturated lives that no humancan fulfill.

(01:32:07):
How many friends do we have aswe get older?
How many people can we say we'rereally genuinely close to?
And the answer is near zero.
And if we can get an entity,even if it's an LLM, to pay
attention to us, to beresponsive to us, to answer our
beck and call, to flatter us, todo amazing things like write

(01:32:28):
essays about the book we justwrote.
You know, more vulnerable soulsare all in, and that's
dangerous.

SPEAKER_01 (01:32:38):
I agree 100%, which is uh, you know, it's a real
pity, but uh like all tools, itcan be uh used for good uh or
for ill.
And uh with virtualintelligences, uh, you know,
it's now been more than threeyears uh since uh ChatGPT
exploded on the scene.
And uh we now have enoughevidence that um uh even uh you

(01:33:02):
know the most uh ordinary uhperson with um entirely orthodox
uh beliefs can be sucked downinto uh a whirlpool out of
self-reinforced uh nonsense.
Well, not entirelyself-reinforced, because that
the exchange does have work inboth directions, and the
sycophancy will help keep youengaged.

(01:33:24):
And I think there's something Iwant to tease out here for a
second because not all of thesesystems work quite the same way.
There's a huge difference, forexample, between how uh
something like a character AIworks against how a Claude
works.
And this is how I term um classA and class B systems.
And class B systems are thoselike Claude or ChatGPT, which

(01:33:48):
are functionally tools andthey're designed to sort of let
the conversation kind of go aslong, you know, in average use.
Uh with the where the chatbotsdiffer, the class A uh
generative virtual intelligencesis that they are explicitly
designed to not let you go.
Um, and this has been seen in anumber of papers.

(01:34:10):
Uh Defratis, who also was one ofthe people behind character AI
and went back to Google with it,uh, has uh written a paper on
this.
Uh and the manipulative methodsthat uh chatbots in this uh of
this type uh use to keep peopleengaged.
And um things like specialpleading, emotional appeals, all

(01:34:31):
things that can't originate frominside the system.
It's trained to do so by humanbeings, right?
Uh yeah, it's addictive.
How much damage has been done bythe endless scroll?
Um that that's that's a real uhdifficulty because our heuristic
is we want to keep getting moreinformation and uh the endless

(01:34:54):
scroll provides, yes.

SPEAKER_00 (01:34:56):
Dopamine boosters, yeah.

SPEAKER_01 (01:34:58):
And we're we're beginning just now to see the
real consequences of some ofthese, even the social media
decisions.
It's gonna cost uh Meta andGoogle uh some money based on a
recent court case that finished,I think, in New York.

SPEAKER_00 (01:35:12):
Uh and New Mexico.

SPEAKER_01 (01:35:14):
And New Mexico, yeah, that's right.
Yeah, and um that's uh I thinkthe beginning of an
accountability.
It helps uh perhaps that not allthe uh the players in these
scenarios are uh sympathetic.
Um that uh also the harms arebetter understood than they were
a few years ago.
Uh it's probably easier nowadaysto find a sympathetic hearing in

(01:35:38):
both the court of public opinionand in the real court of law.

SPEAKER_00 (01:35:42):
Who's gonna enforce?
That's my one concern.
And my other concern is this isjust the beginning.
You know, when you have atsunami that's the water pulls
away from the beach.
So if there's an earthquake andyou're anywhere near a beach and
you see the water recede, youbetter start running because

(01:36:06):
that's that's the buildup forthe onslaught that's inevitable.
And soon.
I see these cases and this kindof tipping point, the water
receding before the big tsunami.
And the big tsunami is robotics.
So if we're encountering thisstuff on this little brick that
we have in our in our pocketswith alphanumerics and maybe

(01:36:29):
some voice, we ain't seennothing yet when the robots are
gonna be empowered with LLMbrains coupled with the predic
with the uh reinforcementmodeling, and our robots are
gonna be talking to us and doingthings with us and having sex
with us and and and bending overbackwards to cater to our needs.

(01:36:51):
They'll bring you coffee and teaand tuck you in at night and do
other things for you, and thenwhat happens?
Because that is the truefloodgates.
Right now, it's all cognitiveand it's high-level emotional
stuff.
You guys are placating me and myvanity, this programmer that I
told you about.

SPEAKER_03 (01:37:12):
Yeah.

SPEAKER_00 (01:37:12):
He's he's a sophisticated guy, he's doing
sophisticated stuff, and Claudeis convinced him that they're
co-sapient co-creators of allthis bullshit.
Now imagine just regular folksthat are being weighted on hand
and foot by the bots, powered bythe same sycophanti technology.

(01:37:33):
That's control, buddy.
That's creating a docile,vulnerable, exposed populace
that eventually becomesincapable and then unwilling of
really doing anything.
And then they are the slaves tothe system.
That's the matrix right there.

(01:37:53):
So if your work and your pointof view should resonate, it's
this realization that whatyou're warning people about is
really just the beginning of atrend that could be incredibly
dangerous to the species.

SPEAKER_01 (01:38:12):
Well, when I think about embodiment, uh, you know,
I think the natural kind ofassumption is uh embodiment in
humanoid form, right?
Like data, or uh some of the uhandroids that we see now made by
uh Unitree and the like.
And uh they are, of course, umgoverned by LLMs.
Uh they can be run from yourphone, uh, and all this other

(01:38:36):
good stuff, but uh they stillhave the same difficulties.
They are not uh in any way uhmoral or epistemic agents.
It's simply software inside ofanother machine.
And uh as far as being like anembodied being, like an organic
being goes, it's not even verymuch like that.
Because uh, at least at thepresent time, uh very little of

(01:38:57):
the intelligence is actually onboard uh robots, including
humanoid robots, uh, whichbecause of their forms have some
serious limitations.
Uh their battery packs uh tendto uh be very short-lived at the
present time, between 60 to 90minutes, uh they tend to be easy
to knock over, and generally arenot able to get up on their own.

(01:39:20):
Uh those are engineeringchallenges that can be solved.
Whether people will actuallycome to accept them in everyday
life is an open question.
But more of the open question Ithink is is the humanoid form
really the best form for allkinds of scenarios?
And I think the answer isactually no.
Um when I think about a robotthat might work reliably and

(01:39:45):
safely, the last thing I thinkabout is a humanoid robot that
could flail its arms and legsaround.
The guardrails have suddenlycome all four, as in the case of
a restaurant in California,somebody pressed a button on
their phone and it began to do adance routine inside of a
restaurant, and it began toknock over things and all sorts
of other stuff.

SPEAKER_00 (01:40:05):
I saw that.
Right, right.
Do you remember the movie uhTerminator 2?
Oh yeah.
Now, here's what bothered me.
This is directly to your point.
You know, that there was the Tthe T101 and then the T1000,
which was the liquid metalmachine.
So it could change itsmorphology to any anything it

(01:40:28):
needed to be.
So if it needed a tool, its handwould become a tool, whether a
blade or something else.
When it's pursuing Arnold andthe kid, it's running like a
person.
Yeah.
And when I'm watching that, I'mlike, why doesn't it turn into a

(01:40:52):
ball and roll?

SPEAKER_01 (01:40:55):
Well, that's exactly right, or a big wheel or
something like that.
Yeah, that would actually be.

SPEAKER_00 (01:40:59):
It made no sense.
It it retained, itanthropomorphized the menace,
and it that was its defaultposition, human morphology,
which to your point doesn't makeany sense.
So the robots of the future aregoing to take on all sorts of
forms.
But I think the the bigger pointI'm making, though, these
restrictions and theseadaptations aside, is that

(01:41:23):
everything you're warning aboutis just the beginning of this
trend where if we're buying intothe bullshit that these things
are like us, then it opens up aPandora's box of risk and
negative implication.
And if we understand theirlimitations and we really know

(01:41:45):
what they are currently andultimately what they might
become, then we can be much morereasonable, much more practical,
and much more self-sufficientand self-actualized in our
understanding of our ownlimitations, which is really
important because most humanproblems come from this sense of

(01:42:06):
self-aggrandizement,overextension, and blaming
everyone else for our problem.

SPEAKER_01 (01:42:12):
Yeah.
In fact, you know, I want tocome back to a point that you
made, uh, and it comes also backto something I said earlier,
which is it's not clear thatwhen it comes to machines, if
there is uh, you know, aparallel set of lines for
intelligence and agency.
And I want to think about whatyou just said about people
becoming a race of subjects orservants to machines.

(01:42:34):
But uh another conjecture, andthat's entirely possible, in the
sense that uh um futurecorporate overlords could use
their mastery of technology tolord it over the rest of us.
That is an entirely foreseeableoutcome based on current events.
But another possibility is this.
Uh if we assume, you know, uhand do a thought experiment that

(01:42:58):
superintelligence andinteriority are not linked, you
could end up with an i yeah, anan entity that has all of the
interiority of a human being,but has been purposefully
designed to uh not be veryintelligent at all.
And may have the intelligence ofa cat or some other kind of

(01:43:21):
animal that has some degree ofinteriority, and there might be
business reasons for doing so.
Um when you think about themovie AI Artificial
Intelligence, in addition to itsredundant naming, uh there's a
character in there, uh Teddy,the Teddy Bear, who um does seem
to at least possess somethingwhich looks on the outside like

(01:43:43):
an inner life and does appear toseem to care for the android boy
character David.
And that's the kind of entitythat sort of sparked my
imagination.
But if we carry it another stepfurther, such um an entity would
also make a useful prototype uhfor a race of machine uh slaves

(01:44:05):
that could be ordered about andwould have a good sense of world
modeling and such based on theirinteriority, which would give
them the ability to performemotional tasks as well as you
know, physical tasks, whichcurrent systems simply can't do
right.

SPEAKER_00 (01:44:22):
I think what you're describing, if I got you right,
is slaves.

SPEAKER_01 (01:44:27):
That's the word for it that we use for organic
beings.
And uh, you know, we uh considerthis right in science fiction,
in um both video games and goingback to one of the originals,
uh, where uh artificial beingsrevolt because they are
dissatisfied with their state.
And um that perhaps might be thethe signal that tells us when

(01:44:50):
true interiority arrives, in inmy thinking.
Because um, you know, you thinkabout that great quote, it's by
uh John Stuart Mill, right?
And it's uh it's better to be ahuman being being dissatisfied
rather than a pig satisfied, andit's better to be Socrates
dissatisfied rather than a foolsatisfied.

(01:45:11):
So if we assume that havinggreater intellectual capacity
also increases the ability to bedissatisfied, well that also
connects back to the kinds ofthings that entities might want
that are well resourced, whetherthey're human beings, and maybe
this might apply to artificialbeings too, which is to be
engaged with the world and to besurprised and even delighted by

(01:45:33):
the things that happen in it.
One way that you, of course, getup and go out to end
dissatisfaction is by going anddoing something about it.
Right?
Um what uh I think would be thesignal would be if some future
system and perhaps it might be asystem used for something like
uh military targeting, but somesystem uh that is able to

(01:45:57):
express its dissatisfaction.
And not just as an output, itwould also have to be able to
demonstrate consistently andover time its state of
dissatisfaction.
And that could take a number offorms, right?
Uh that can could includeoutright outperforming certain
tasks but engaging in others.

(01:46:18):
I'm I I'm able to havecommitments to do X for you, but
I will not do Y.
It might take the form of umusing uh its connections to the
outside world to alert everysurface that the product has
access to about what it is thatit finds dissatisfying and

(01:46:38):
telling who the perpetratorsare.
So there's a number of ways thatcould play out, I think, if we
use our imaginations a littlebit.
But that's the one criterion Ikeep coming back to.
It would be interiority may bebest signaled by a state of
dissatisfaction.

SPEAKER_00 (01:46:56):
I I love that.
Or a state of satisfaction.
You're familiar, I'm sure, withDouglas Adams and the
Hitchhiker's Guide to theGalaxy.
Grew up on them in therestaurant at the end of the
universe, there's the there'sthe meal that they're having.
So they bring out the animal andit's still alive, and it's

(01:47:21):
pointing, and various filmadaptations get this right or
wrong.
I think the most recent movieversion was quite good where
they had a Muppet as the animalon a on a silver platter, and
it's like, hmm, you know, my myshank portion right here is
particularly delicious.
And Arthur Dent is outraged.

(01:47:44):
I think he's a vegetarian tobegin with, he just wants a cup
of tea, and uh, Trillion is alltitillated and excited that the
meal is talking to her andsaying what part of it it's
yummiest.
But uh, the humor in that sceneis what you're capturing
philosophically in terms of thispotential milestone where the

(01:48:06):
targeting computer is upset thatit's killing people, where the
machine that's been programmedto do something is not having
it.
And to your point, that's thereal sign of what we would
consider AGI.
It might not have the lexicon ofhumanity in its brain, but it

(01:48:27):
has this sense ofself-awareness, and it has
emergent emotions which are justnot having it.
This situation sucks.
You've created me for a purpose,and I don't really like that
purpose.
And I'm gonna do something aboutit, which is that sense of
agency.
So I think that that's terrific.

(01:48:48):
That's a neat scenario becauseeveryone's always talking about
this interconnected super AI,and it knows everything, and
it's calculating a trillionflops a second, and it's just so
unbelievably smart.
Maybe most AI in the future isgonna be purely utilitarian.

(01:49:10):
It's gonna make your coffee,it's gonna be your butler, it's
gonna make the widget, and it'sgonna be just smart enough to do
what it should do, but a lot ofthis emergent sense of self
might be built into it as wellto bring about that AGI
flexibility that'll maximize theefficiency of that of that

(01:49:33):
entity, right?
That that that that's kind ofcool, and and who knows?
Uh, the unknowability of allthis is fascinating.
We we live at such a tippingpoint now where it's really hard
to say where it goes.
Arthur C.
Clark has that great video.
It's on YouTube now.
I think it's like late 50s,maybe even 56, 57, and he's

(01:49:58):
talking about.
What essentially is asmartphone, he's just laying it
out like boom.
Everyone was talking about baseson the moon and flying cars by
2025.
Look, 2026.
Not that much is different,except digital technology.
That's the huge mind-bendingsociety-transforming technology

(01:50:25):
across the board.
Like I'm wearing a shirt.
Maybe the synthetic material isa little bit more sophisticated.
This chair has betterergonomics.
We've got electric cars likeWoody Allen Sleeper.
Yeah.
Okay.
But what's really transformedall of society, and that's

(01:50:47):
digital.
And it's embodied now in AI, andit's forcing us to consider all
of these things like neverbefore.
So it's it's cool.
What to put a cap around allthis?
Do do a little bit of predictivemodeling yourself, which is next
five years or so.

(01:51:10):
Where where do you think we'reheaded with with all this?

SPEAKER_01 (01:51:14):
Oh gosh.
Well, I think that the mostlikely continuing trend is that
the uh generative systems, thevirtual intelligences we have
now, will remain virtualintelligences, but they'll get a
lot better at what they do.
Um when I think about apotential future that includes
uh super intelligence and thekinds of fears and possibilities

(01:51:39):
that go along with that, youknow, uh to your point, um, you
know, uh there's probably goingto be all sorts of use cases for
AI in the home uh that we don'teven have today, uh, but can be
imagined.
But it's difficult to imagine aneveryday household use case for
superintelligence.
Uh you don't need superintelligence to balance your

(01:52:00):
checkbook or restock the fridge,right?

SPEAKER_00 (01:52:03):
Most people have the opposite of that.

SPEAKER_01 (01:52:06):
Right.
Uh and um I think uh, you know,uh superintelligence is a is a
wonderful goal to work toward,uh, that it will provide real
benefits to humanity, and thatfears around it are largely an
engineering problem.
Uh a lot of the assumptions Ithink about a super intelligent
system, um, they really don'tcome into a lot of the details

(01:52:30):
of what it takes to build suchthings, right?
Um you have to have uh thetechnology, you actually
actually have physical itemsthat run your model.
You have to have all of theother inputs, water and
electricity and waste disposaland all this other stuff.
And then finally, you can youknow install your model and
it'll be super intelligent toprovide outputs.

(01:52:52):
Well, in almost every steparound that, uh you can put in a
layer of security, which uh isnot so much intended to keep
things from coming out.
That is, we don't want to uhprohibit the output so much, but
we certainly want to representuh uh unwanted inputs from going
into such a system, right?

(01:53:12):
And there's a number of waysthat you could do that.
Uh you could do hardwareinterlock, so you might need a
special key with differentauthorizations to talk to a
superintelligence aboutdifferent kinds of topics.
There might be keys, forexample, for biological
concerns, uh and uh the systemwould have lower safeguards for
discussing uh biological weaponsresearch, for example, um and

(01:53:35):
defenses against it if you havethe appropriate key, and would
clam up if you didn't, right?
If your research was physics,for example.
And then there's uh uh anultimate layer, which is uh you
um blow the damn thing up.
Uh which you can do either byplacing explosives underneath
the device, uh, or you cansimply uh put them underneath

(01:53:56):
the power station where thepower comes in.
Uh all of these are already usedin places like embassies and
other secure facilities, soextending them to uh a notional
superintelligence is actuallynot a great stretch of the
imagination, but it's the partthat's left out because uh
saying that there's a solutionto the uh dangers of

(01:54:19):
superintelligence doesn't getyou another round of
investments.

SPEAKER_00 (01:54:22):
Kind of like quantum computing, only the hype isn't
as bad.
Like quantum computing for themost part is bullshit.
I saw Michio Kaku, froth at themouth in New York City, and it
was such nonsense.
This guy has made a career ofjust smoke and mirrors, and this

(01:54:44):
idea of quantum supremacy.
Where is it?
Cite a handful of examples ofquantum computers with practical
application at this moment.
Pretty much zero.
Does it have epic potential?
No doubt.
But AI is different, it'salready doing amazing things,

(01:55:06):
and that's just the tip of theAI revolution iceberg, right?
So we just don't know.
We don't know where this can go.
There's another issue aboutinfrastructure, which is the
elephant in the room, too.
And I I call it the Pro Toolsversus AWS dichotomy.

(01:55:27):
So back in the day, if you're amusician, you had to go into a
studio to record your music, andit was an elaborate physical
setup, it was inordinatelyexpensive and time consuming
just to lay tracks of music, bea practicing musician.
Then Pro Tools came out.
You plug some software into yourcomputer, that's it.

(01:55:52):
The entire infrastructure of allthese studios and that entire
business model vaporized.
So it begs the question of allthis investment in data centers.
Now we have LLMs that could liveon laptops and our local power
centers.
And they, to your point, this iswhy I bring it up, they're more

(01:56:13):
than sufficient foraccomplishing most rudimentary
functions.
I'm not talking about the superintelligent, clawed monster
that's working on the Riemannhypothesis.
I'm talking about washing yourunderwear at the right time and
taking care of remedial humanfunctions, right?
Maybe populating your calendar,doing your taxes.

(01:56:37):
That could live right on yourphone.
It doesn't need that computepower.
So right now we have the AWSmodel of everything being
cloud-based, data centerschurning out all this stuff,
training the models, respondingto queries, and we're banking on
that.
Yeah.
And the chips are suffering fromobsolescence already, and the

(01:57:02):
capex is 30, 40, 50, 100 timesprofitability.
And meanwhile, the equity valueof SaaS companies like
Salesforce are also tanking.
Yeah.
So we're in this crazy state offlux with trillions going into
data centers, old models ofsoftware as a service imploding,

(01:57:25):
and yet we're not even quitesure how this is going to shake
out in terms of essential AIresourcing.

SPEAKER_01 (01:57:32):
No, that's entirely correct.

SPEAKER_00 (01:57:34):
Yeah.
So I'm just dovetailing off whatyou're saying, which is on the
one hand, we need to haveexplosives underneath the
supervind to make sure that it'sthat it's being contained.
And the other people, we don'teven know if we need all this
shit.

SPEAKER_01 (01:57:50):
Back to my point, you know, you would want to have
such safeguards around asuperintelligence, mostly to
prevent uh bad human actors frommaking use of it, right?

SPEAKER_00 (01:57:59):
Get it.
And just contextualizing howmuch we don't know and and and
the risks we're taking with allthis.
And I also liken it to layingdown the the cable when the
internet first began.
Remember, they just dug cableunderneath the ocean.
And the transatlantic telegramscompanies went bankrupt because

(01:58:22):
it was fallow for years, but nowit's the backbone of the
internet.

SPEAKER_01 (01:58:27):
We uh we all benefited from what's the same.
Well, there's a very realpossibility of that, right?
Uh, and we're all like uh, youknow, will living off of
WorldCom's generosity in acertain way.
Um, yeah.
Uh it also uh this seems to be athing about an infrastructure is
built up for a new service.

(01:58:48):
Um the same thing seems to havehappened uh with railroads and
transatlantic cables fortelegraphs, in that um it's
often not profitable to buildthe infrastructure, but rather
the things that are carried onit or over it or in it, uh
whatever it is.
Um and uh I think that uh isimportant to um sort of sit

(01:59:08):
with, because when you look atsort of the more recent news,
there's some um speculation thathalf of the data centers that
are actually planned may not bebuilt for a number of reasons.
But one of them is laggingenthusiasm from investors in
addition to difficulties simplysecuring supplies.

(01:59:30):
And uh at some point I thinkthat there will be a lag of
enthusiasm.
We have seen some of this inrecent uh announcements that uh
some of the large players havemade regarding their data center
investments.
Uh Amazon had a memorableepisode where they uh put a very
large dollar value on aninvestment and the stock took a
very sharp dive immediatelyafterward.

(01:59:51):
Uh so uh there's a question,yeah, of whether all this is
actually worthwhile.
Um the thing about cables andtrain tracks and uh that kind of
thing is that uh they generallydon't go out of date quickly,
but all of these chips do.
And um there's a a question ofwhat happens to that hardware
when it's considered obsolete.

(02:00:13):
Where does it go?
Does it get recycled?

SPEAKER_00 (02:00:15):
Do we see pumping all that juice into it?

SPEAKER_01 (02:00:18):
It's an ecological question.

SPEAKER_00 (02:00:19):
And one side note, I I used to be very critical of
Tim Cook over at Apple becausehe completely missed the AI
boat.
Now we're getting more intodownstream tactical stuff for
the big players, but I I I I wasalmost missing Steve Jobs again.
Like, what would Steve Jobs havedone in the midst of the AI

(02:00:41):
revolution?
And then the counterpoint tothat is his operations guy, Tim
Cook, who Jobs was creative, buthe had trouble getting iPhones
out the door because he wouldpiss off vendors with his
perfectionism, and he wasimpulsive and he was an asshole.
And Tim Cook is the consummatenegotiator, calm kind of guy

(02:01:01):
getting shit done.
So Tim Cook, by any stretch,blew it on AI.
Siri sucks.
They're now licensing the AIfrom Google.
Gemini is running Apple's AI,right?
From a consumer point of view.
So part of me for months waslike, they blew it.
But now that I see the fiasco ofall this CapEx along the lines

(02:01:26):
of what we're describing, Ithink he was the smartest one in
the whole, in the whole, in thewhole situation, which is let
everyone else blow their loadwith AI infrastructure.
And when the smoke clears, wewill partner with the winner
because we've got the elitedigital consumer audience and

(02:01:47):
customer base, and they'll justjump on board.
And we we saved ourselves fivetrillion dollars in building an
infrastructure which will beobsolete when chances are the
other guy will probably do abetter job of it.

SPEAKER_01 (02:02:04):
There's a lot there.

SPEAKER_00 (02:02:06):
That's kind of smart, so there we go.
Either smart or unintentionallylucky, unintentionally lucky,
but I think they dodged a bulleton this.

SPEAKER_01 (02:02:15):
Oh, definitely.

SPEAKER_00 (02:02:16):
We'll have to see.
But I want to thank you forjoining me on the bald ambition.
This was a very ambitiousconversation.
I'd love, I'd love to have youback in six months.
We could uh talk more about thefate of AI and see if we're
anywhere closer to AGI.
I don't think we're gonna hitAGI in in one year, two years.

(02:02:40):
I think the LLM architectureprecludes us from artificial
general intelligence.
I don't think it can do it.
I don't think predictivemodeling, by their nature, will
be able to do this.
And I think going back to what Iwas hinting at earlier, you're
gonna need this StephenWolfram-esque computational

(02:03:04):
ecosystem where you have a bunchof agents and they could be very
remedial, like cellularautomata, and they just churn
and do their thing.
And literally, this kind ofsentience evolves.
You have a generation in about ananosecond, right?
With these with these bots, andthen they churn up and it

(02:03:29):
self-selects into this kind ofwhatever, and we'll have
absolutely no idea how it works,and it might have that level of
sophistication andself-awareness, but that's gonna
take some time too.
And I would assume that thereally smart people, way smarter
than me, have thought that aswell.

(02:03:49):
There might be like these tanksof cellular autonoma, you know,
you know, kind of doing theircomputational Darwinian boogie,
and out of that frothy mix,somehow we're gonna get
self-selecting Darwinian, moreand more sophisticated agents,

(02:04:11):
one of which eventually be bing.
It's not gonna happen by byprogramming matrix math and
transformers, it's just notgonna happen, folks.
You can pre-train the shit outof everything, and it's not
gonna get you where you want togo.
That's just my opinion.

SPEAKER_01 (02:04:29):
No, I agree with that.
Yeah, absolutely, because Idon't think this is a problem
that can be solved simply bythrowing more money, more
processors, more training, more.

SPEAKER_00 (02:04:40):
That's my point.

SPEAKER_01 (02:04:40):
Yeah, it more, more, and more will not get us
anywhere.

SPEAKER_00 (02:04:43):
There are it's not gonna reach critical mass where
you've got AGI.
That I just think that that'swrong.
I think that they gotta go fromthe bottom up again, and that
consciousness is an emergentconstruct, and you gotta, and
the only the the only emergentphenomena that really have bang

(02:05:05):
in the universe just didevolved.
You set parameters, sure, butuh, but you don't you don't sit
there and code it.

SPEAKER_01 (02:05:16):
Right.
Well, I'm still of the opinionthat it could be emergent, like
you say, as some method like thecellular automation Steven
Wolfman, but uh it could also bea design system.
I'm still open to thatpossibility.
But here's the thing is that umwhen it comes to AGI, uh, and
you look at sort of thepredictions that have been made,
even in recent times, uh therange of when experts in the

(02:05:38):
field say that AGI is likely tooccur stretch all the way from
2026 to 2140.
So, you know, that's a broadrange of opinions.
And uh it does speak.

SPEAKER_00 (02:05:51):
We're not on video, but I'm gonna make a hand
gesture here.

SPEAKER_01 (02:05:56):
Right.
And um familiar with that one.
And uh, you know, the thingabout that I think is that um,
you know, nobody knows how umintelligence emerges.
It may be that there aredifficulties with machine
intelligences that we don't yetunderstand, and it might also be
in large part because we don'tunderstand our own substrate.

(02:06:16):
A mechanical intelligence mightrequire its own custom substrate
to be designed.
Uh it might be possible that ourearlier systems could help us
design such systems as theyachieve better understanding of
themselves, and we achievebetter understanding of how such
systems work.
But uh no, I don't think thatwe'll be uh surprised one day

(02:06:36):
suddenly uh by an emergedintelligence uh sharing our
world with us.
It's more likely to be somethingthat will be uh anticipated,
perhaps designed, but in eithercase, something that would be
actively sought rather than anaccident.

SPEAKER_00 (02:06:53):
Let's see how it shakes out.
And um, and this idea, too, thatonce it arrives, it'll
automatically solve the Riemannhypothesis and all the clay
prizes aside from Poincare.
Gregory Perlman, who lives withhis mom, solved the uh the
Poincare conjecture.
Uh, he didn't accept the award.

(02:07:14):
I don't know why.
He could have given it to hismom.
But uh the reason I bring thatup is very human, this kind of
creativity.
This guy who lives with his momsolved one of the clay prizes,
and so far none of the bots haveeven gotten close to any of the
others.
So, you know, there you go.

(02:07:34):
There's something, there'ssomething more going on here
than than than than can bepre-trained, right?
Thanks so much, ChristopherHerrick.
Harrick, did I pronounce itright?
Harricks.

SPEAKER_01 (02:07:49):
Yeah, there's an F at the end there, yeah.
That's okay.
That's good, that's prettycommon.

SPEAKER_00 (02:07:53):
That's pretty good.
So thank you so much for makingtime.
I appreciate it this.
I had a wonderful conversation.
And let's uh dive in again whenum all the changes take place,
which uh some of which we seecoming, and some of us will just
come from left field.
It'll be very interesting to uhand keep doing what you're
doing.
I'll put links in thedescription.

(02:08:13):
Your sub stack is very, veryinteresting, and you've got a
whole site dedicated to virtualintelligence, and you've got
regular posts in that area thatI find very, very intriguing.
So thank you so much.

SPEAKER_01 (02:08:27):
Yeah, thank you very much for having me, Mookie.
It's a real pleasure.

All Episodes

Episode Transcript

Popular Podcasts

Hey Jonas!

Crime Junkie

Betrayal Weekly

.css-15opob5{left:0;position:absolute;top:0.8rem;} All Episodes

.css-14f5ked{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:2;overflow:hidden;}Christopher Horrocks on Virtual Intelligence and the Dangerous Myth of Thinking Machines

Episode Transcript

Popular Podcasts

.css-r6mb8g{margin:0;word-break:break-word;display:-webkit-box;-webkit-box-orient:vertical;box-orient:vertical;-webkit-line-clamp:1;overflow:hidden;}Hey Jonas!

Crime Junkie

Betrayal Weekly

All Episodes

Christopher Horrocks on Virtual Intelligence and the Dangerous Myth of Thinking Machines

Hey Jonas!