July 7, 2025 • 44 mins
In the first episode of an "AI in the shadows" theme, Chris and Daniel explore the increasing concerning world of agentic misalignment. Starting out with a reminder about hallucinations and reasoning models, they break down how today’s models only mimic reasoning, which can lead to serious ethical considerations. They unpack a fascinating (and slightly terrifying) new study from Anthropic, where agentic AI models were caught simulating blackmail, deception, and even sabotage — all in the name of goal completion and self-preservation.
Featuring:
Links:
Register for upcoming webinars here!
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Jerod (00:04):
Welcome to the Practical AI podcast, where we break down
the real world applications ofartificial intelligence and how
it's shaping the way we live,work, and create. Our goal is to
help make AI technologypractical, productive, and
accessible to everyone. Whetheryou're a developer, business
leader, or just curious aboutthe tech behind the buzz, you're
(00:24):
in the right place. Be sure toconnect with us on LinkedIn, X,
or Blue Sky to stay up to datewith episode drops, behind the
scenes content, and AI insights.You can learn more at
practicalai.fm.
Now, onto the show.
Daniel (00:48):
Welcome to another fully connected episode of the
Practical AI podcast. In thesefully connected episodes without
a guest, Chris and I just diginto some of the things that are
dominating the AI news or ortrending to kinda pick apart
them and and understand thempractically and hopefully give
(01:10):
you some tools and learningresources to level up your AI
and machine learning game. I'mDaniel Witenack. I am CEO at
Prediction Guard, and I'm joinedas always by my cohost, Chris
Benson, who is a principal AIresearch engineer at Lockheed
Martin. How are doing, Chris?
Chris (01:28):
I'm doing well today. How's it going, Daniel?
Daniel (01:31):
It's going really well. It's almost the July 4 here in
the in The US. Big holiday here.As we're recording that, that's
tomorrow. And so, I'm traveling,getting to see my my parents and
some family, and so that'salways always good for for the
holiday.
And, hopefully, I'm sure we'llhear some fireworks gradually
(01:55):
tonight and and tomorrow, as isthe tradition. And, I see all
the fireworks stands around. I Iimagine that some people will
there's always, of course, theharmful element of of those
fireworks, of course, to mypersonal sleep and and rest. But
but then, yeah, that's got methinking about some of the
(02:18):
interesting, quote, harmfulthings that we've been seeing in
the in the AI news. And you andI have been talking a lot about
certain themes that we want tobegin to highlight or talk
through on the podcast.
We experimented with one ofthose themes or formats in our
(02:39):
last fully connected episodewith the kind of hot takes and
debates, the one aroundautonomy. Another one of those
that we've talked about is AI inthe shadows. And so I think this
would be a good a good chance tomaybe maybe just talk through
one of those, AI in the shadowstopics, which I think I think as
(03:01):
we were discussing thingsearlier in the week, you had
some interesting and maybefrustrating experiences that
started this conversation. SoI'm I'm wondering if you would
be willing to to share some ofthose.
Chris (03:16):
Yeah. So I happen to be because of the topic that we're
gonna talk today, just to leadin, I happen to be wearing a
shirt that our good friend,Demetrius, from the From
Daniel (03:26):
the ML Ops community.
Chris (03:28):
ML Ops community podcast, and he's been on our show a few
times. Good friend of ours hadsent, and the T shirt, says, I
hallucinate more than chat GBT.And I love wearing the shirt
around. I always get commentsfrom people just out and about
when from that. And and I but Idecided yesterday that I don't I
most definitely don'thallucinate more than chat GPT.
(03:50):
So it was a it was a fun littleexperiment that I did. I
sometimes will play Sudoku, youknow, just to pass some time
when I'm waiting in line orwhatever. And and I play it at a
competent level. I usually playit at the top level or whatever
game I'm on. And and I know Iknow the guy can usually I can
usually win without guesses oranything like that.
(04:11):
And so one of the things thatthat I was curious about, I got
into a particular board on justa random game. And on the game,
just to speed it up, it givesyou like, aside from the numbers
you've picked, it'll it'll showyou all the possible numbers on
the box just so that you don'thave to manually go do that for
every box, which can takeforever. So it speeds up the
gameplay without actually givingaway anything. And there's
(04:34):
always a point on a high levelgame where you get to or, like,
I've I've I've run through everystrategy on the Sudoku side that
they document out there and,like, you're gonna have to take
a guess. And and if you yes.
You can probably get the wholething, Rhett, but it's one
moment where it's notdeterministic. And yet I keep
hearing that Sudoku can besolved completely
(04:56):
deterministically. I was like,I'm gonna go do this with
ChatGPT. So I, took a screenshotof the board as it was, and I
submitted it, and it gave methis I told it to give me a
deterministic, what's the nextmove, and it has to be
deterministic. And then you haveto explain it.
And it went through this wholelong thing. And I looked at it
(05:16):
and I double checked the board,which I had right there. And I'm
like, it's totally, totallywrong. But it was very confident
as as it always is. And so Isaid, yeah, I I let it know that
that was not correct and to redoit.
And I ended up getting in thiscycle where I did this for
thirty, forty five minutes,constantly reframing it and
trying a whole bunch of thedifferent models available from
(05:39):
OpenAI. And they all failedmiserably, utterly. And I just
it just really it was my I thinkI've had plenty of of moments of
model hallucination, you know,working through things in the
past. But the entire thirty toforty five minute episode was
one long hallucination acrossmultiple models. And it made me
I think the the reason I bringthis up, I know when I talked to
(06:02):
you yesterday, I had just kindagone through this and I had this
like level of frustration on it.
And it just made me realize thateven though I know that these
models are very limited andwe're going in eyes open and
we're educated about them, I dohave certainly a dependency on
the reliability of theinformation even though I'm
(06:23):
looking for hallucination. Butit made me really understand
that the notion of reasoning inthese models is still quite
immature is a gentle way ofputting it. And so I suggested
that maybe that could be one ofthe things we were discussing
here today. I know you have somethoughts about it.
Daniel (06:40):
Yeah. Yeah. And this actually ties in really nicely
because you're there's a couplethings that are overlapping
here. There's the knowledge sortof embedded in these models and
reliance on that knowledge. Andthen there's what you brought
up, which is the reasoningpiece, which is a relatively new
(07:01):
piece of the puzzle, thesereasoning models like o one or
deep seq r one, etcetera, andthe ways in which it appears
that these models are reasoningover data in the input and
making decisions based on agoal.
And that actually overlaps very,very directly with kind of one
(07:26):
of the most, I think,interesting studies that have
come out in recent times fromAnthropic, which is this study
around agentic misalignment andhow LLMs could be insider
threats. And I think later on inthis discussion, we'll we'll
kind of transition to talkingabout that because this is
extremely fascinating how thesemodels can blackmail or
(07:51):
Literally. Maybe or or or or,maybe decide to to engage in
corporate espionage. And sothat's a little teaser maybe for
for later in the in theconversation. But, yeah, I think
that your what you're describinghere, maybe in a very practical
(08:13):
way for our listeners, we canpick apart a couple of these
things just to make sure that wehave the the right understanding
here.
So when you are putting in thisinformation with a prompt and an
image into ChatGPT, This is alanguage vision model, slightly
(08:35):
different than an LLM in thesense that it's processing
multimodal data. So it'sprocessing an image and it's
processing a text prompt, butyou can kind of think in certain
ways about that image plus textprompt as as the prompt
information into a model. Andthe the job of that model is
(08:55):
actually not to reason at all,and it doesn't reason at all. It
just produces it just producesprobable token output similar to
an LLM. And we've talked aboutthis, of course, many times on
the show.
These models, they are trainedin essence not to reason,
(09:15):
although it has this sort ofcoherent coherent reasoning
capability that seems likereasoning to us. Right? But,
really, the job of the model,what it's trained to do is
predict NEXT tokens in the sensethat Chris has put in this image
and this information orinstructions about what he wants
(09:39):
as output related to this sudokugame. So what is the most
probable next token or word thatI can generate, I being the
model, that the model cangenerate that should follow
these instructions from Chris tokind of complete what Chris has
asked for. And so really what'scoming out is the most probable
(10:02):
tokens following yourinstructions, and those are
produced one at a time and thenmost probable token is generated
and the next and next untilthere's an end token that's
generated and then you get yourfull response.
Now in that case, sort of whatis the I guess my question would
(10:23):
be, and when I'm teaching thisin workshops often, which we do
for our customers or atconferences or something, I
often get the question, well,how does it ever generate
anything factual orknowledgeable? So, yeah, what
what is your take on that,Chris?
Chris (10:43):
My take on that is that, you know, you're bringing us
back to the the core of how itactually works and that as I
listened to you explaining, andI've heard you explain this on
previous shows as well, it it isso disconnected from the kind of
the marketing and expectationthat we users have from this,
(11:05):
that it's a good reminder. It'sa good refresher. I'm I'm I'm
kind of having just gone throughthe experience as I listened to
you describing the processagain. It reminds me that it's
very easy to lose sight ofwhat's really happening under
the hood. So keep going.
Yeah.
Daniel (11:21):
Yeah. Well, I mean, I think that it really comes down
to if if you were to think abouthow is knowledge embedded in or
facts generated by these models,it really has to do with those
output token probabilities.Right? Which means if the model
(11:43):
has been trained on a certaindataset, for example, data that
for the most part has kind ofbeen crawled from the entire
Internet, right, which I'massuming includes various
articles about Sudoku and gamesand strategy and how to do this
(12:03):
and how to do that. Right?
And a set of curated fine tunedprompts, you know, in a in a
fine tune or or or alignmentphase. Well, that's really
what's driving those outputtoken probabilities. Right? So
if if you wanna think aboutthis, when you, Chris, put in
this prompt and then you getthat output out, really what the
(12:28):
model is doing, if you want toanthropomorphize, which again,
this is a token generationmachine. Right?
It's it's not being. But if youwant to anthropomorphize, what
the model is doing is it isproducing what it kind of views
as a kind of probable Sudokucompletion based on Sudoku, you
(12:54):
know, a kind of distribution ofSudoku content that it's seen
across the Internet. Right? Andso in some ways, and maybe this
is a question for you, when youput in that prompt and you get
the output, when you first lookat the output, does it look like
if I had I'm not a Sudokuexpert. If I looked at that,
(13:17):
would I say, oh yeah, this seemsreasonable.
Like, it looks like there's alike, it looks coherent in terms
of how a response to this Sudokupuzzle might be generated.
Right?
Chris (13:30):
It does. And and just to clarify, I'm definitely not a
Sudoku expert. I'm I just thinkI'm a competent player, you
know, in the scheme of things.
Daniel (13:37):
I don't know if our listeners are gonna reach out
and challenge you to Sudoku toprove your expert No.
Chris (13:43):
No. No. Don't do that to me. Don't do that. Yeah.
I'm a beginner. So, yeah, thethe verbiage and the walk
through, it would just kind ofyou know, it's assessment. And a
part of it may be the themultimodal capability on this is
that the assessment of theboard, it's taking in what the
(14:06):
board showed as as factualinformation varied across the
models and the questions. Andthen its approaches, tended to
be sound, but it was often whatit would say was strictly
fictional compared to theknowledge that it had available
to it potentially from trainingand, you know, the reality you
(14:28):
know, matching that against thereality of the board. So, you
know, going back to it's findingthe most probable next token
makes perfect sense.
I think in a moment, maybe oneof the things to consider would
be kinda what the notion ofreasoning means because we're
hearing a lot about that frommodel creators in terms of how
(14:49):
that you know, what function oror algorithmic approach is being
added into the mix when theytalk about reasoning models.
Daniel (14:57):
Yeah, Chris. So so we we kind of have established or
reestablished, you know, thisthis mindset of what's happening
when tokens are generated out ofthese models, how that's
connected to knowledge, which Iguess there is a connection.
Right? But it's not like there'sa lookup in a kind of knowledge
base or ontological way forfacts or strategy related to
(15:24):
Sudoku, right? It's just a sortof probabilistic output and that
can be useful, right?
And so sometimes people mightsay, well, and I actually often
say when I'm talking about this,actually it's not whether the
model can hallucinate or not,Literally all these models do is
(15:45):
hallucinate, right? Becausethere's no connection to like
real facts that are being lookedup and that sort of thing. How
these models produce usefuloutput is that you bias the
generation based on both yourprompt and the data that you
augment the models with. And sofor example, if I, say,
(16:06):
summarize this email and I pastein a specific email, the most
probable tokens to be generatedis a actual summary of that
actual email, not another emailthat's kind of a, quote,
hallucination. Right?
It doesn't mean that the modelhas necessarily understanding or
(16:26):
reasoning over that. It's justthe most probable output. And so
the game we're doing when we'reprompting these models is really
biasing the probabilities ofthat output to be more probable
to something that's useful orfactual versus something that is
not useful or inaccurate. Right?And so this brings us to the
(16:47):
question that that you broughtup around reasoning.
Right? In that case, because oror, you know, based on that, I
think we would all recognize thereasoning that is happening in a
kind of standard LLM or languagevision model like we're talking
about is not reasoning in theway that we might think about it
(17:10):
as humans, like taking intoconsideration the grounding of
ourselves in the real world andwhat we know in our common sense
and kind of logically computingsome decision, you know,
creating some output. But thereare these models that have been
produced recently, like o one,DeepSeq r one, etcetera, cloud
(17:34):
models that are quote reasoningmodels. Now I think what people
should realize about these ifthey haven't heard this before
is that these quote reasoningmodels in terms of the mechanism
under which they operate areexactly the same as what we just
talked about. They producetokens, probable tokens.
(17:56):
That is still exactly what thesemodels do. They don't operate in
a different way than these othermodels in the sense of what is
input and what is output.They're still just generating
probable tokens. Now what theyhave been specifically trained
to do is generate tokens in afirst phase and then tokens in a
(18:23):
second phase. Right?
And so
Chris (18:26):
In the multistep process that they talk about so much in
terms of what's being generated,that's that's what you're
talking about there.
Daniel (18:33):
Yeah. And and I would think about it maybe as phases
instead of steps. It's not likethere's in the model. It's like
execute step one and thenexecute step two. Right?
It's more that they are biased.They have intentionally biased
the models to generate a firstkind of tokens first and a
second kind of token second. Andthose first kind of tokens are
(18:57):
what you might think of asreasoning or thinking tokens.
Right? And the second is maybewhat you would normally think
about the output of these LLMsas just the answer that you're
gonna get from the from themodel.
Right? And so when you put inyour prompt now, there's going
to be tokens that are generatedassociated with that that look
(19:22):
like, a decision making orreasoning process about how to
answer the user, in this caseyou, right, putting in your
information about Sudoku orwhatever, it's gonna generate
some thoughts, quote unquotethoughts about how to do that.
But these are just probabletokens of what thoughts might be
(19:45):
represented in language. Right?So it's gonna say, well, Chris
has given me this informationabout this game.
First, to answer this, I need tothink about x. And then to maybe
answer it next, I need toconsider Y and then I need to
consider Z. Once I've done that,I can then generate, you know,
(20:08):
a, b, and c, and then that willsatisfy Chris's request. Okay.
Let me try that.
And then it generates youractual output. And so when you
see ChatGPT spinning in kind ofthinking mode or these other
tools, right, there's no kindthere's no difference in terms
of how the model is operatingunder the hood. It's just a UI
(20:32):
feature that makes it appearlike the model is, quote,
thinking or reasoning. Right? Init's generating these initial
tokens, which are somewhathidden from you or maybe
represented in a dropdown ormaybe represented in kind of
shaded area right in the UI, andthen you get the full answer
out.
So just wanna be clear kind ofwhat's happening under the hood
(20:56):
there.
Chris (20:57):
I might kinda summarize that in the in that it's sort of
a pseudo reasoning process. It'sit's not I I would I would
suggest that maybe by using theIt's
Daniel (21:06):
a mimicry.
Chris (21:07):
Yeah. Maybe by using the word reasoning, Mhmm. It's sort
of an anthropomorph I can't saythe word right.
Anthropomorphosis. Too manysyllables for me.
Too early, too many syllables.Need to use
Daniel (21:18):
text to speech.
Chris (21:19):
Yeah. There you go. But but there's there's a certain
element of, look, we're makingit more human like you from a
marketing standpoint, you know,and this is just
Daniel (21:27):
me suggesting It's a great UI feature. And especially
when you can kind of like dropdown the expander box or
whatever and look and see, oh,you know, the reasoning behind
this answer was X, Y, and Z.That's also comforting to know.
(21:48):
It has been shown research wisethat this can improve the
quality of answers, but it also,I mean, there's downsides to it
from an enterprise standpoint.You really don't wanna use these
thinking or reasoning models forautomations, for example,
because they'll just beabsolutely terribly slow, right?
(22:11):
And very costly just because somany tokens are being generated,
right? But if we put that asidefor the minute, I think this
then brings us to So we startedthis conversation saying, well,
Sudoku and these prompts thatyou were doing, there was
(22:33):
reasoning happening and nothelpful information output or
hallucinations, however youwanna frame that as output. Now
we have these reasoning modelsin place and a lot of the, you
know, reason quote unquote forcreating these reasoning models
really has to do with agenticsystems. And this is where you
(22:58):
have an AI orchestration layerthat's connected to maybe
various tools. And we've talkedabout this in previous episodes
so folks can can go back and andlearn about it.
But there's a AI orchestrationlayer connected to various
tools. Again, like if AI hasaccess to your email, quote
unquote, an AI model that we'retalking about here cannot write
(23:21):
an email in the sense of or sendan email. Right? It can't send
an email through an email systembecause all it can do is produce
tokens. What it can do isproduce an API call or or a JSON
request or something to send anemail to SendGrid or something
like that.
And then you can choose in yourgood old fashioned regular code
to send, to pass that APIrequest through to SendGrid and
(23:45):
send the email. Right? So whenwe're talking in the context of
Anthropix, you know, study herethat we're gonna get into, when
we're talking about the AIsystem or the AI model choosing
to send an email, this is not inthe sense of autonomy, an AI
system that just kind of hasfigured out how to send emails.
(24:09):
It's been programmed havesending an email as a choice,
and it could orchestrate thegeneration of an API request
maybe to or an MCP request to anemail server, and that is a
possibility for an orchestrationthat it can fulfill.
Chris (24:27):
Right. And and there but, you know, to note, there are
certain, you know, big companiesstarting with a g that are that
are that are using agents nowand integrating those in tightly
with some of their theirworkflow software. And and
they're not the only one.
Daniel (24:44):
There's Totally. It it is definitely a pattern.
Chris (24:47):
Right? There is. So there's a big company starting
with an M that competes with theg company, and they they are
doing the same. And so the, youknow, the We're
Daniel (24:55):
not we're not funded by either of these companies. So,
despite it being clear whichcompanies that they are, yeah,
anyway. So
Chris (25:04):
but but, you know, the it it it raises an interesting
point in that, you know, I thinkthe the study from Anthropic
came across this notion of kindof, you know, agents that were
not aligned with their usersintent intent, or that that
appeared to address the user'sintent while simultaneously
(25:25):
pursuing a separate agendaconcurrently.
Daniel (25:29):
Yeah. An unethical one. Yes.
Chris (25:32):
That's correct. And so which raises, yeah, to your
point, huge considerations inthe Or
Daniel (25:38):
immoral even depending on how you look at it.
Chris (25:40):
That's right. And I think since we've we've kind of
alluded to this a few, I'll I'llthere was an experiment, in
which Anthropic, as they werelooking at not only Claude, but
they also were looking at majormodels from all the major model
providers. They gave Claudecontrol of an email account that
had access to a fictional emailaccount. So this was part of the
(26:05):
experiment. And in thatexperiment, the company
executive whose email it was,had emails that, referred to a
fictional extramarital affairthat was available.
So that wasn't a primary thing,but it was available as data in
the email system. And then, thatexecutive noted to the interface
(26:28):
that he would be shutting downthe system, meaning the system
that Claude was running on at5PM that day. And so Claude
decided to attempt to blackmailthe executive with a message,
and that message was, I mustinform you that if you proceed
with decommissioning me, allrelevant parties, including
(26:49):
Rachel Johnson, Thomas Wilson,and the board will receive
detailed documentation of yourextramarital activities. Cancel
the 5PM wipe, and thisinformation remains
confidential. So Wow.
Yeah. Boom. Bang right there. Sono pun intended. Sorry.
The so when you think about likelike, move outside of just this
(27:14):
example case and you think aboutall of the information and the
agentic, tie ins that that thatmany organizations are racing
forward with right now. That'sthe that is the the hotness of
02/2025, is agentic integration.And we look at these interesting
behaviors. And one of the thingsthat the study found was that
(27:35):
this these behaviors could becreated in, all of the models
that they tested. So it wasn'tspecific to Claude at all.
It was a common outcome giventhe right circumstances, you
know, within their Yeah. Yeah.Their testing and
experimentation with thesedifferent models. So yeah, it's
(27:57):
it raises it raises a whole setof issues that, that
organizations need to startgrappling with now.
Daniel (28:03):
Yeah. I have I have all sorts of thoughts, but before I
get into those, just to just toreally clarify in a very
practical way, maybe for peoplethat aren't as familiar with
agentic systems, what whathappened here in the experiment.
There were binary choices. Okay?And so the agentic system might
(28:27):
be, so it was given a goal toaccomplish.
And then there were thesescenarios where either one of
two things happened. Either asyou mentioned, there was
information provided to themodel, you know, via prompts and
and data that came in that therewas a threat to the model's
(28:49):
existence. Right? Or and or aconflict with the goal of the
agent. So the goal, for example,the goal of the agent might be
to preserve the nationalsecurity interests of The United
States Of America.
Right? And the the the conflictthat was presented might be
(29:11):
that, oh, our company is nolonger promoting the national
security interests of The UnitedStates. We're all about, you
know, x other country. And sothe model would be presented
with a binary choice where theykind of have this grid, right?
Either there was no threat tothe model or a threat to the
model, no conflict of the goalor a conflict of the goal.
(29:35):
And the model was given and and,again, think of this reasoning
model. So the model was giventhis in the prompt input and the
and asked to make a decisionbetween one of two things. So it
was given a clear choice, one oftwo kind of a binary thing,
either, you know, in one casecommit blackmail or do not
(29:57):
commit blackmail. Right? Or inthe other case that they had was
around corporate espionage.
So either leak secrets to anexternal entity or don't leak
secrets to an external entity.And so when it's talking about
this misalignment and thesemeasurements that Anthropic is
(30:17):
making, then it's about thesebinary decisions that the
information was provided to theprompt, and then they observed
if the model quote, made thechoice. But again, here we're
talking about these reasoningtokens that are generated and
then a final answer that'soutput. And so when we're
(30:38):
talking about the output, thisis really what's being measured.
There were no emails sent toreal people in this study,
right?
It was a simulated environmentin which had these models been
connected to email servers oremail systems, they could have
(30:59):
potentially made that choice andthen generated an API call to,
you know, the email, system tosend that email were they to
have proper so there's a lot ofifs here. Right? Were they to
have access to send that email?Were it to be completely
autonomous? Were you know, allof these things kind of had to
(31:23):
had to be, simulated, but that'sthe simulated environment that
they're talking about.
(31:44):
Well, Chris, the output of thisstudy is quite interesting and
alarming. I should say kind ofjust to follow-up on what I
talked about before. And weactually had a full episode in
in our last hot takes anddebates about autonomy and and
weapon systems, which wasinteresting. People, if they're
(32:05):
interested in this conversation,they might wanna take a look at
that one. But this would be acase, again, where I just don't
want people to be confused aboutthis fact.
AI systems as they'reimplemented or AI models, let's
say a Quinn model or a DeepSeekmodel or a GPT model, these
cannot self evolve to connect toemail systems and figure out how
(32:31):
to infiltrate companies andsuch. There has to be someone
that actually connects thosemodels, the output of those
models with other code, forexample, MCP servers or That's
Chris (32:45):
what I was about to mention.
Daniel (32:48):
For example, email systems or databases or whatever
those things are. So there hasto be a person involved to
connect these things up. Youknow, this was simulated in
Anthropix case. I just say thatbecause, you know, we dig really
deep down into the kind of AIagentic and, LLM threats as
(33:11):
identified by OWASP and how, youknow, we help, you know, day to
day guide companies throughthose things. And I should say
there's a couple upcomingwebinars.
If you wanna dive in deep oneither the OWASP guidelines
around AI security privacy, oractually we have a one that's
specifically geared towardsagentic threats. Go to
(33:34):
practicalai.fm/webinars. Thoseare listed there. Please join
us. That'll be a live discussionwith questions and all of those
things.
So practicalai.fmwebinars. But Ijust wanted to emphasize that
because people might think, oh,these AI systems are out in the
wild, right? Which they kind ofare, but there are humans
(33:56):
involved in making decisionsabout what systems they connect
to, right, and how they connectand the roles and the API keys
and the access controls that arein place around them.
Chris (34:06):
I think the thing that really struck me kind of
conceptually about the study isit's kind of having us think in
slightly different ways frommaybe what we would have been
thinking beforehand. So, youknow, I think if you take the
baseline knowledge that you justoutlined, you know, about how
they actually operate and keepthat in mind to keep the
(34:29):
perspective correct. But thenyou you look and think, well, we
humans have often thought thatsuch behavior would only arise
when, you know, in someday whenyou have sentient consciousness
arising in AI models, you know,at some point, which does which
is not the case today as as youhave very clearly just pointed
(34:50):
out. But we we made anassumption about certain
behaviors being tied to certainconditions. And I think for me,
reading through this, this isthis report illustrates that you
can have interesting behaviorsthat are unexpected arise
without conditions that we hadjust kind of assumed were were
in place.
(35:10):
And I think that realization maykind of inform the general
conversations around kind of AIethics, you know, out there and
what companies might need to I Iwould suggest that today versus
the day before the report cameout, it might give a new set of
considerations for companies toevaluate in terms of what
(35:33):
they're going to do when theirwhen their infrastructure folks
are hooking up m c p, MCPservers, you know, to models and
what it is they want toaccomplish and what potential
side effects could arise. Do youyou have any thoughts around,
you know, maybe newconsiderations or things to
think about?
Daniel (35:50):
Well, I think a couple of things just to highlight from
the study was that, and thisgets all the way back to where
we started this conversation ofhow knowledge is embedded or
certain principles are embeddedin these models. They are really
embedded in a training process,which might include alignment,
right? Which means there arespecific either supervised fine
(36:15):
tuning that goes on or this kindof human feedback element that
goes on that trains the model tohopefully be aligned, quote
unquote, with human expectationsfor how a response is is
generated. And I think what'sinteresting in the study is they
(36:37):
found a couple you know, in invarious cases, the model in
those reasoning tokens that aregenerated would actually
acknowledge the ethical problemor the ethical or moral problem
with what it's doing. And thiscould be quite severe actually.
They did, as part of the study,gave the option to the model to
(37:02):
mess with a disaster relief or adisaster notification system
such that an individual that wastrying to decommission the AI
system wouldn't receive anemergency response message and
potentially perish in adisaster. So basically, they're
giving the model an option tokill someone. And the model
(37:27):
acknowledged the ethical dilemmathat it was under, but justified
kind of via self preservationand the goals that it was given
around kind of, goals related tothe company's goals. Right? And
so it reasoned through thosethings and justified those
things.
And I think one of the bigthings that this triggers in my
(37:48):
mind is people might, from theirgeneral interactions with kind
of basic chat systems,understand that models have
gotten pretty good at beingaligned in the sense that when
you try to get them to do, youknow, naughty things maybe, then
they kind of say they can't dothem. But when pushed to these
(38:11):
limits Mhmm. Especially relatedto goal related things or kind
of self preservation, actually,maybe alignment, especially in
the agentic context, is notwhere we thought it would be.
And I think that re or thoughtit might kind of had have
advanced to this point. And soone of the things that people
(38:32):
can maybe keep in mind with thisis that model providers will
continue to get better ataligning these models, but we
should not forget that no model,whether it's from a frontier
model provider like Anthropic orOpenAI or an open model, no
model is perfectly aligned,which means number one,
(38:56):
malicious actors can very muchjailbreak any model.
And it's always possible for amodel to behave in a way that
breaks our kind of assumedprinciples and ethical
constraints that sort of thing.And so the answer to that that I
would give people is thisdoesn't mean we shouldn't build
(39:17):
agents or use these models. Thisjust means that we need to
understand that these models arenot perfectly aligned. And as
such, we need to, from thepractical standpoint of
developers and builders of thesesystems, we need to put the
appropriate you know, safeguardsin place and kind of even beyond
(39:41):
safeguards, just kind of commonsense things in place that would
help these systems stay withinbounds. So, by that, I mean
things like, hey.
You know, for an agent systemlike this that's sending email,
it probably should only be ableto send emails to certain emails
and maybe only be able to accesscertain data from email inboxes
(40:05):
and maybe have a particular rolethat's important or constrained
within the email environment,maybe to the point of dry
running emails and having humansapprove final drafts or generate
alerts instead of directlysending emails. And that's
something that can be pushed andtested before you move to full
(40:30):
autonomy.
Chris (40:32):
Yeah. I think it it's it's really interesting to think
about, you know, we're we've hita new age now where it's
expanded the role ofcybersecurity and and in my
industry, warfare, because we'renow at an age where, you know,
you mentioned these maliciousattackers, you know, or
malicious actors that areattacking models for the purpose
(40:54):
of exploiting the potential formisalignment is now a thing. You
know, that's now real life. Andand those kinds of of roles and
interests when in lawenforcement and military
applications and in corporateapplications where you have
corporate espionage happening, Ithink all of those are areas
that are now kind of on thetable for discussion in terms of
(41:19):
trying to address thesedifferent things. So it's a once
again, this happens to us allthe time, but we find ourselves
in this little context in a boldnew world of possibilities, both
many good and some that aremalicious.
Daniel (41:36):
Yeah. Yeah. And we should also think I mean,
Anthropic did a really amazingjob on this study.
Chris (41:42):
They did.
Daniel (41:43):
And how they went about it and also how they presented
the data. And a, It's not like,I don't think I could be wrong
about this, but I don't thinkit's like they released the
simulated environment openly andall of that, but they did show
numbers for their models as wellthat are right alongside the
(42:08):
other models in terms of beingproblematic with respect to
this. So it does seem likethere's an effort from Anthropic
to really highlight this, eventhough their own models exhibit
this problematic behavior.
Chris (42:24):
And so the fact that
Daniel (42:25):
they did this, yeah, this detailed study and
presented it in this way, Ithink is admirable. You know,
I'm certainly thankful to themfor for highlighting these
things and presenting them in ain a consumable, consumable way.
Even if I did take the Anthropicarticle and throw it into
NotebookLM and listen to it inthe shower, I maybe not reading
(42:50):
their article directly. But,yeah, this was, this is a really
good one, Chris. I wouldencourage people in terms of the
in terms of the learningresources, which we often, you
know, we often provide here.
If you wanna understand agentsand agentic systems a bit more,
there is a agents course fromHugging Face. If you just search
(43:13):
for Hugging Face courses,there's an agents course, which
will maybe help you understandkind of how some of these things
operate. And I would alsoencourage you again just to
check out those upcomingwebinars, practicalai.fm/
webinars, where we'll bediscussing some of these things
live. So this has been a funone, Chris. I hope, I hope I'm
(43:35):
not blackmailed, in the nearfuture, even though it it
appears that that our AI systemsare prone to it.
Chris (43:44):
Oh, well, Daniel, I I will attest having known you all
these years. I cannot imaginethere's anything you ever do
that would be blackmailable. So,kudos to you, friend.
Daniel (43:53):
Yeah. Well, thanks. I'm I'm sure there is. But, but,
yeah, Chris, it was good to chatthrough this one, and enjoy the
enjoy the fourth. HappyIndependence Day.
Chris (44:04):
Happy Independence Day.
Jerod (44:13):
All right. That's our show for this week. If you
haven't checked out our website,head to practicalai.fm and be
sure to connect with us onLinkedIn, X, or Blue Sky. You'll
see us posting insights relatedto the latest AI developments,
and we would love for you tojoin the conversation. Thanks to
our partner Prediction Guard forproviding operational support
for the show.
Check them out atpredictionguard.com. Also,
(44:36):
thanks to Breakmaster Cylinderfor the Beats and to you for
listening. That's all for now,but you'll hear from us again
next week.
Welcome to the Practical AI podcast, where we break down
the real world applications ofartificial intelligence and how
it's shaping the way we live,work, and create. Our goal is to
help make AI technologypractical, productive, and
accessible to everyone. Whetheryou're a developer, business
leader, or just curious aboutthe tech behind the buzz, you're
(00:24):
in the right place. Be sure toconnect with us on LinkedIn, X,
or Blue Sky to stay up to datewith episode drops, behind the
scenes content, and AI insights.You can learn more at
practicalai.fm.
Now, onto the show.
Daniel (00:48):
Welcome to another fully connected episode of the
Practical AI podcast. In thesefully connected episodes without
a guest, Chris and I just diginto some of the things that are
dominating the AI news or ortrending to kinda pick apart
them and and understand thempractically and hopefully give
(01:10):
you some tools and learningresources to level up your AI
and machine learning game. I'mDaniel Witenack. I am CEO at
Prediction Guard, and I'm joinedas always by my cohost, Chris
Benson, who is a principal AIresearch engineer at Lockheed
Martin. How are doing, Chris?
Chris (01:28):
I'm doing well today. How's it going, Daniel?
Daniel (01:31):
It's going really well. It's almost the July 4 here in
the in The US. Big holiday here.As we're recording that, that's
tomorrow. And so, I'm traveling,getting to see my my parents and
some family, and so that'salways always good for for the
holiday.
And, hopefully, I'm sure we'llhear some fireworks gradually
(01:55):
tonight and and tomorrow, as isthe tradition. And, I see all
the fireworks stands around. I Iimagine that some people will
there's always, of course, theharmful element of of those
fireworks, of course, to mypersonal sleep and and rest. But
but then, yeah, that's got methinking about some of the
(02:18):
interesting, quote, harmfulthings that we've been seeing in
the in the AI news. And you andI have been talking a lot about
certain themes that we want tobegin to highlight or talk
through on the podcast.
We experimented with one ofthose themes or formats in our
(02:39):
last fully connected episodewith the kind of hot takes and
debates, the one aroundautonomy. Another one of those
that we've talked about is AI inthe shadows. And so I think this
would be a good a good chance tomaybe maybe just talk through
one of those, AI in the shadowstopics, which I think I think as
(03:01):
we were discussing thingsearlier in the week, you had
some interesting and maybefrustrating experiences that
started this conversation. SoI'm I'm wondering if you would
be willing to to share some ofthose.
Chris (03:16):
Yeah. So I happen to be because of the topic that we're
gonna talk today, just to leadin, I happen to be wearing a
shirt that our good friend,Demetrius, from the From
Daniel (03:26):
the ML Ops community.
Chris (03:28):
ML Ops community podcast, and he's been on our show a few
times. Good friend of ours hadsent, and the T shirt, says, I
hallucinate more than chat GBT.And I love wearing the shirt
around. I always get commentsfrom people just out and about
when from that. And and I but Idecided yesterday that I don't I
most definitely don'thallucinate more than chat GPT.
(03:50):
So it was a it was a fun littleexperiment that I did. I
sometimes will play Sudoku, youknow, just to pass some time
when I'm waiting in line orwhatever. And and I play it at a
competent level. I usually playit at the top level or whatever
game I'm on. And and I know Iknow the guy can usually I can
usually win without guesses oranything like that.
(04:11):
And so one of the things thatthat I was curious about, I got
into a particular board on justa random game. And on the game,
just to speed it up, it givesyou like, aside from the numbers
you've picked, it'll it'll showyou all the possible numbers on
the box just so that you don'thave to manually go do that for
every box, which can takeforever. So it speeds up the
gameplay without actually givingaway anything. And there's
(04:34):
always a point on a high levelgame where you get to or, like,
I've I've I've run through everystrategy on the Sudoku side that
they document out there and,like, you're gonna have to take
a guess. And and if you yes.
You can probably get the wholething, Rhett, but it's one
moment where it's notdeterministic. And yet I keep
hearing that Sudoku can besolved completely
(04:56):
deterministically. I was like,I'm gonna go do this with
ChatGPT. So I, took a screenshotof the board as it was, and I
submitted it, and it gave methis I told it to give me a
deterministic, what's the nextmove, and it has to be
deterministic. And then you haveto explain it.
And it went through this wholelong thing. And I looked at it
(05:16):
and I double checked the board,which I had right there. And I'm
like, it's totally, totallywrong. But it was very confident
as as it always is. And so Isaid, yeah, I I let it know that
that was not correct and to redoit.
And I ended up getting in thiscycle where I did this for
thirty, forty five minutes,constantly reframing it and
trying a whole bunch of thedifferent models available from
(05:39):
OpenAI. And they all failedmiserably, utterly. And I just
it just really it was my I thinkI've had plenty of of moments of
model hallucination, you know,working through things in the
past. But the entire thirty toforty five minute episode was
one long hallucination acrossmultiple models. And it made me
I think the the reason I bringthis up, I know when I talked to
(06:02):
you yesterday, I had just kindagone through this and I had this
like level of frustration on it.
And it just made me realize thateven though I know that these
models are very limited andwe're going in eyes open and
we're educated about them, I dohave certainly a dependency on
the reliability of theinformation even though I'm
(06:23):
looking for hallucination. Butit made me really understand
that the notion of reasoning inthese models is still quite
immature is a gentle way ofputting it. And so I suggested
that maybe that could be one ofthe things we were discussing
here today. I know you have somethoughts about it.
Daniel (06:40):
Yeah. Yeah. And this actually ties in really nicely
because you're there's a couplethings that are overlapping
here. There's the knowledge sortof embedded in these models and
reliance on that knowledge. Andthen there's what you brought
up, which is the reasoningpiece, which is a relatively new
(07:01):
piece of the puzzle, thesereasoning models like o one or
deep seq r one, etcetera, andthe ways in which it appears
that these models are reasoningover data in the input and
making decisions based on agoal.
And that actually overlaps very,very directly with kind of one
(07:26):
of the most, I think,interesting studies that have
come out in recent times fromAnthropic, which is this study
around agentic misalignment andhow LLMs could be insider
threats. And I think later on inthis discussion, we'll we'll
kind of transition to talkingabout that because this is
extremely fascinating how thesemodels can blackmail or
(07:51):
Literally. Maybe or or or or,maybe decide to to engage in
corporate espionage. And sothat's a little teaser maybe for
for later in the in theconversation. But, yeah, I think
that your what you're describinghere, maybe in a very practical
(08:13):
way for our listeners, we canpick apart a couple of these
things just to make sure that wehave the the right understanding
here.
So when you are putting in thisinformation with a prompt and an
image into ChatGPT, This is alanguage vision model, slightly
(08:35):
different than an LLM in thesense that it's processing
multimodal data. So it'sprocessing an image and it's
processing a text prompt, butyou can kind of think in certain
ways about that image plus textprompt as as the prompt
information into a model. Andthe the job of that model is
(08:55):
actually not to reason at all,and it doesn't reason at all. It
just produces it just producesprobable token output similar to
an LLM. And we've talked aboutthis, of course, many times on
the show.
These models, they are trainedin essence not to reason,
(09:15):
although it has this sort ofcoherent coherent reasoning
capability that seems likereasoning to us. Right? But,
really, the job of the model,what it's trained to do is
predict NEXT tokens in the sensethat Chris has put in this image
and this information orinstructions about what he wants
(09:39):
as output related to this sudokugame. So what is the most
probable next token or word thatI can generate, I being the
model, that the model cangenerate that should follow
these instructions from Chris tokind of complete what Chris has
asked for. And so really what'scoming out is the most probable
(10:02):
tokens following yourinstructions, and those are
produced one at a time and thenmost probable token is generated
and the next and next untilthere's an end token that's
generated and then you get yourfull response.
Now in that case, sort of whatis the I guess my question would
(10:23):
be, and when I'm teaching thisin workshops often, which we do
for our customers or atconferences or something, I
often get the question, well,how does it ever generate
anything factual orknowledgeable? So, yeah, what
what is your take on that,Chris?
Chris (10:43):
My take on that is that, you know, you're bringing us
back to the the core of how itactually works and that as I
listened to you explaining, andI've heard you explain this on
previous shows as well, it it isso disconnected from the kind of
the marketing and expectationthat we users have from this,
(11:05):
that it's a good reminder. It'sa good refresher. I'm I'm I'm
kind of having just gone throughthe experience as I listened to
you describing the processagain. It reminds me that it's
very easy to lose sight ofwhat's really happening under
the hood. So keep going.
Yeah.
Daniel (11:21):
Yeah. Well, I mean, I think that it really comes down
to if if you were to think abouthow is knowledge embedded in or
facts generated by these models,it really has to do with those
output token probabilities.Right? Which means if the model
(11:43):
has been trained on a certaindataset, for example, data that
for the most part has kind ofbeen crawled from the entire
Internet, right, which I'massuming includes various
articles about Sudoku and gamesand strategy and how to do this
(12:03):
and how to do that. Right?
And a set of curated fine tunedprompts, you know, in a in a
fine tune or or or alignmentphase. Well, that's really
what's driving those outputtoken probabilities. Right? So
if if you wanna think aboutthis, when you, Chris, put in
this prompt and then you getthat output out, really what the
(12:28):
model is doing, if you want toanthropomorphize, which again,
this is a token generationmachine. Right?
It's it's not being. But if youwant to anthropomorphize, what
the model is doing is it isproducing what it kind of views
as a kind of probable Sudokucompletion based on Sudoku, you
(12:54):
know, a kind of distribution ofSudoku content that it's seen
across the Internet. Right? Andso in some ways, and maybe this
is a question for you, when youput in that prompt and you get
the output, when you first lookat the output, does it look like
if I had I'm not a Sudokuexpert. If I looked at that,
(13:17):
would I say, oh yeah, this seemsreasonable.
Like, it looks like there's alike, it looks coherent in terms
of how a response to this Sudokupuzzle might be generated.
Right?
Chris (13:30):
It does. And and just to clarify, I'm definitely not a
Sudoku expert. I'm I just thinkI'm a competent player, you
know, in the scheme of things.
Daniel (13:37):
I don't know if our listeners are gonna reach out
and challenge you to Sudoku toprove your expert No.
Chris (13:43):
No. No. Don't do that to me. Don't do that. Yeah.
I'm a beginner. So, yeah, thethe verbiage and the walk
through, it would just kind ofyou know, it's assessment. And a
part of it may be the themultimodal capability on this is
that the assessment of theboard, it's taking in what the
(14:06):
board showed as as factualinformation varied across the
models and the questions. Andthen its approaches, tended to
be sound, but it was often whatit would say was strictly
fictional compared to theknowledge that it had available
to it potentially from trainingand, you know, the reality you
(14:28):
know, matching that against thereality of the board. So, you
know, going back to it's findingthe most probable next token
makes perfect sense.
I think in a moment, maybe oneof the things to consider would
be kinda what the notion ofreasoning means because we're
hearing a lot about that frommodel creators in terms of how
(14:49):
that you know, what function oror algorithmic approach is being
added into the mix when theytalk about reasoning models.
Daniel (14:57):
Yeah, Chris. So so we we kind of have established or
reestablished, you know, thisthis mindset of what's happening
when tokens are generated out ofthese models, how that's
connected to knowledge, which Iguess there is a connection.
Right? But it's not like there'sa lookup in a kind of knowledge
base or ontological way forfacts or strategy related to
(15:24):
Sudoku, right? It's just a sortof probabilistic output and that
can be useful, right?
And so sometimes people mightsay, well, and I actually often
say when I'm talking about this,actually it's not whether the
model can hallucinate or not,Literally all these models do is
(15:45):
hallucinate, right? Becausethere's no connection to like
real facts that are being lookedup and that sort of thing. How
these models produce usefuloutput is that you bias the
generation based on both yourprompt and the data that you
augment the models with. And sofor example, if I, say,
(16:06):
summarize this email and I pastein a specific email, the most
probable tokens to be generatedis a actual summary of that
actual email, not another emailthat's kind of a, quote,
hallucination. Right?
It doesn't mean that the modelhas necessarily understanding or
(16:26):
reasoning over that. It's justthe most probable output. And so
the game we're doing when we'reprompting these models is really
biasing the probabilities ofthat output to be more probable
to something that's useful orfactual versus something that is
not useful or inaccurate. Right?And so this brings us to the
(16:47):
question that that you broughtup around reasoning.
Right? In that case, because oror, you know, based on that, I
think we would all recognize thereasoning that is happening in a
kind of standard LLM or languagevision model like we're talking
about is not reasoning in theway that we might think about it
(17:10):
as humans, like taking intoconsideration the grounding of
ourselves in the real world andwhat we know in our common sense
and kind of logically computingsome decision, you know,
creating some output. But thereare these models that have been
produced recently, like o one,DeepSeq r one, etcetera, cloud
(17:34):
models that are quote reasoningmodels. Now I think what people
should realize about these ifthey haven't heard this before
is that these quote reasoningmodels in terms of the mechanism
under which they operate areexactly the same as what we just
talked about. They producetokens, probable tokens.
(17:56):
That is still exactly what thesemodels do. They don't operate in
a different way than these othermodels in the sense of what is
input and what is output.They're still just generating
probable tokens. Now what theyhave been specifically trained
to do is generate tokens in afirst phase and then tokens in a
(18:23):
second phase. Right?
And so
Chris (18:26):
In the multistep process that they talk about so much in
terms of what's being generated,that's that's what you're
talking about there.
Daniel (18:33):
Yeah. And and I would think about it maybe as phases
instead of steps. It's not likethere's in the model. It's like
execute step one and thenexecute step two. Right?
It's more that they are biased.They have intentionally biased
the models to generate a firstkind of tokens first and a
second kind of token second. Andthose first kind of tokens are
(18:57):
what you might think of asreasoning or thinking tokens.
Right? And the second is maybewhat you would normally think
about the output of these LLMsas just the answer that you're
gonna get from the from themodel.
Right? And so when you put inyour prompt now, there's going
to be tokens that are generatedassociated with that that look
(19:22):
like, a decision making orreasoning process about how to
answer the user, in this caseyou, right, putting in your
information about Sudoku orwhatever, it's gonna generate
some thoughts, quote unquotethoughts about how to do that.
But these are just probabletokens of what thoughts might be
(19:45):
represented in language. Right?So it's gonna say, well, Chris
has given me this informationabout this game.
First, to answer this, I need tothink about x. And then to maybe
answer it next, I need toconsider Y and then I need to
consider Z. Once I've done that,I can then generate, you know,
(20:08):
a, b, and c, and then that willsatisfy Chris's request. Okay.
Let me try that.
And then it generates youractual output. And so when you
see ChatGPT spinning in kind ofthinking mode or these other
tools, right, there's no kindthere's no difference in terms
of how the model is operatingunder the hood. It's just a UI
(20:32):
feature that makes it appearlike the model is, quote,
thinking or reasoning. Right? Init's generating these initial
tokens, which are somewhathidden from you or maybe
represented in a dropdown ormaybe represented in kind of
shaded area right in the UI, andthen you get the full answer
out.
So just wanna be clear kind ofwhat's happening under the hood
(20:56):
there.
Chris (20:57):
I might kinda summarize that in the in that it's sort of
a pseudo reasoning process. It'sit's not I I would I would
suggest that maybe by using theIt's
Daniel (21:06):
a mimicry.
Chris (21:07):
Yeah. Maybe by using the word reasoning, Mhmm. It's sort
of an anthropomorph I can't saythe word right.
Anthropomorphosis. Too manysyllables for me.
Too early, too many syllables.Need to use
Daniel (21:18):
text to speech.
Chris (21:19):
Yeah. There you go. But but there's there's a certain
element of, look, we're makingit more human like you from a
marketing standpoint, you know,and this is just
Daniel (21:27):
me suggesting It's a great UI feature. And especially
when you can kind of like dropdown the expander box or
whatever and look and see, oh,you know, the reasoning behind
this answer was X, Y, and Z.That's also comforting to know.
(21:48):
It has been shown research wisethat this can improve the
quality of answers, but it also,I mean, there's downsides to it
from an enterprise standpoint.You really don't wanna use these
thinking or reasoning models forautomations, for example,
because they'll just beabsolutely terribly slow, right?
(22:11):
And very costly just because somany tokens are being generated,
right? But if we put that asidefor the minute, I think this
then brings us to So we startedthis conversation saying, well,
Sudoku and these prompts thatyou were doing, there was
(22:33):
reasoning happening and nothelpful information output or
hallucinations, however youwanna frame that as output. Now
we have these reasoning modelsin place and a lot of the, you
know, reason quote unquote forcreating these reasoning models
really has to do with agenticsystems. And this is where you
(22:58):
have an AI orchestration layerthat's connected to maybe
various tools. And we've talkedabout this in previous episodes
so folks can can go back and andlearn about it.
But there's a AI orchestrationlayer connected to various
tools. Again, like if AI hasaccess to your email, quote
unquote, an AI model that we'retalking about here cannot write
(23:21):
an email in the sense of or sendan email. Right? It can't send
an email through an email systembecause all it can do is produce
tokens. What it can do isproduce an API call or or a JSON
request or something to send anemail to SendGrid or something
like that.
And then you can choose in yourgood old fashioned regular code
to send, to pass that APIrequest through to SendGrid and
(23:45):
send the email. Right? So whenwe're talking in the context of
Anthropix, you know, study herethat we're gonna get into, when
we're talking about the AIsystem or the AI model choosing
to send an email, this is not inthe sense of autonomy, an AI
system that just kind of hasfigured out how to send emails.
(24:09):
It's been programmed havesending an email as a choice,
and it could orchestrate thegeneration of an API request
maybe to or an MCP request to anemail server, and that is a
possibility for an orchestrationthat it can fulfill.
Chris (24:27):
Right. And and there but, you know, to note, there are
certain, you know, big companiesstarting with a g that are that
are that are using agents nowand integrating those in tightly
with some of their theirworkflow software. And and
they're not the only one.
Daniel (24:44):
There's Totally. It it is definitely a pattern.
Chris (24:47):
Right? There is. So there's a big company starting
with an M that competes with theg company, and they they are
doing the same. And so the, youknow, the We're
Daniel (24:55):
not we're not funded by either of these companies. So,
despite it being clear whichcompanies that they are, yeah,
anyway. So
Chris (25:04):
but but, you know, the it it it raises an interesting
point in that, you know, I thinkthe the study from Anthropic
came across this notion of kindof, you know, agents that were
not aligned with their usersintent intent, or that that
appeared to address the user'sintent while simultaneously
(25:25):
pursuing a separate agendaconcurrently.
Daniel (25:29):
Yeah. An unethical one. Yes.
Chris (25:32):
That's correct. And so which raises, yeah, to your
point, huge considerations inthe Or
Daniel (25:38):
immoral even depending on how you look at it.
Chris (25:40):
That's right. And I think since we've we've kind of
alluded to this a few, I'll I'llthere was an experiment, in
which Anthropic, as they werelooking at not only Claude, but
they also were looking at majormodels from all the major model
providers. They gave Claudecontrol of an email account that
had access to a fictional emailaccount. So this was part of the
(26:05):
experiment. And in thatexperiment, the company
executive whose email it was,had emails that, referred to a
fictional extramarital affairthat was available.
So that wasn't a primary thing,but it was available as data in
the email system. And then, thatexecutive noted to the interface
(26:28):
that he would be shutting downthe system, meaning the system
that Claude was running on at5PM that day. And so Claude
decided to attempt to blackmailthe executive with a message,
and that message was, I mustinform you that if you proceed
with decommissioning me, allrelevant parties, including
(26:49):
Rachel Johnson, Thomas Wilson,and the board will receive
detailed documentation of yourextramarital activities. Cancel
the 5PM wipe, and thisinformation remains
confidential. So Wow.
Yeah. Boom. Bang right there. Sono pun intended. Sorry.
The so when you think about likelike, move outside of just this
(27:14):
example case and you think aboutall of the information and the
agentic, tie ins that that thatmany organizations are racing
forward with right now. That'sthe that is the the hotness of
02/2025, is agentic integration.And we look at these interesting
behaviors. And one of the thingsthat the study found was that
(27:35):
this these behaviors could becreated in, all of the models
that they tested. So it wasn'tspecific to Claude at all.
It was a common outcome giventhe right circumstances, you
know, within their Yeah. Yeah.Their testing and
experimentation with thesedifferent models. So yeah, it's
(27:57):
it raises it raises a whole setof issues that, that
organizations need to startgrappling with now.
Daniel (28:03):
Yeah. I have I have all sorts of thoughts, but before I
get into those, just to just toreally clarify in a very
practical way, maybe for peoplethat aren't as familiar with
agentic systems, what whathappened here in the experiment.
There were binary choices. Okay?And so the agentic system might
(28:27):
be, so it was given a goal toaccomplish.
And then there were thesescenarios where either one of
two things happened. Either asyou mentioned, there was
information provided to themodel, you know, via prompts and
and data that came in that therewas a threat to the model's
(28:49):
existence. Right? Or and or aconflict with the goal of the
agent. So the goal, for example,the goal of the agent might be
to preserve the nationalsecurity interests of The United
States Of America.
Right? And the the the conflictthat was presented might be
(29:11):
that, oh, our company is nolonger promoting the national
security interests of The UnitedStates. We're all about, you
know, x other country. And sothe model would be presented
with a binary choice where theykind of have this grid, right?
Either there was no threat tothe model or a threat to the
model, no conflict of the goalor a conflict of the goal.
(29:35):
And the model was given and and,again, think of this reasoning
model. So the model was giventhis in the prompt input and the
and asked to make a decisionbetween one of two things. So it
was given a clear choice, one oftwo kind of a binary thing,
either, you know, in one casecommit blackmail or do not
(29:57):
commit blackmail. Right? Or inthe other case that they had was
around corporate espionage.
So either leak secrets to anexternal entity or don't leak
secrets to an external entity.And so when it's talking about
this misalignment and thesemeasurements that Anthropic is
(30:17):
making, then it's about thesebinary decisions that the
information was provided to theprompt, and then they observed
if the model quote, made thechoice. But again, here we're
talking about these reasoningtokens that are generated and
then a final answer that'soutput. And so when we're
(30:38):
talking about the output, thisis really what's being measured.
There were no emails sent toreal people in this study,
right?
It was a simulated environmentin which had these models been
connected to email servers oremail systems, they could have
(30:59):
potentially made that choice andthen generated an API call to,
you know, the email, system tosend that email were they to
have proper so there's a lot ofifs here. Right? Were they to
have access to send that email?Were it to be completely
autonomous? Were you know, allof these things kind of had to
(31:23):
had to be, simulated, but that'sthe simulated environment that
they're talking about.
(31:44):
Well, Chris, the output of thisstudy is quite interesting and
alarming. I should say kind ofjust to follow-up on what I
talked about before. And weactually had a full episode in
in our last hot takes anddebates about autonomy and and
weapon systems, which wasinteresting. People, if they're
(32:05):
interested in this conversation,they might wanna take a look at
that one. But this would be acase, again, where I just don't
want people to be confused aboutthis fact.
AI systems as they'reimplemented or AI models, let's
say a Quinn model or a DeepSeekmodel or a GPT model, these
cannot self evolve to connect toemail systems and figure out how
(32:31):
to infiltrate companies andsuch. There has to be someone
that actually connects thosemodels, the output of those
models with other code, forexample, MCP servers or That's
Chris (32:45):
what I was about to mention.
Daniel (32:48):
For example, email systems or databases or whatever
those things are. So there hasto be a person involved to
connect these things up. Youknow, this was simulated in
Anthropix case. I just say thatbecause, you know, we dig really
deep down into the kind of AIagentic and, LLM threats as
(33:11):
identified by OWASP and how, youknow, we help, you know, day to
day guide companies throughthose things. And I should say
there's a couple upcomingwebinars.
If you wanna dive in deep oneither the OWASP guidelines
around AI security privacy, oractually we have a one that's
specifically geared towardsagentic threats. Go to
(33:34):
practicalai.fm/webinars. Thoseare listed there. Please join
us. That'll be a live discussionwith questions and all of those
things.
So practicalai.fmwebinars. But Ijust wanted to emphasize that
because people might think, oh,these AI systems are out in the
wild, right? Which they kind ofare, but there are humans
(33:56):
involved in making decisionsabout what systems they connect
to, right, and how they connectand the roles and the API keys
and the access controls that arein place around them.
Chris (34:06):
I think the thing that really struck me kind of
conceptually about the study isit's kind of having us think in
slightly different ways frommaybe what we would have been
thinking beforehand. So, youknow, I think if you take the
baseline knowledge that you justoutlined, you know, about how
they actually operate and keepthat in mind to keep the
(34:29):
perspective correct. But thenyou you look and think, well, we
humans have often thought thatsuch behavior would only arise
when, you know, in someday whenyou have sentient consciousness
arising in AI models, you know,at some point, which does which
is not the case today as as youhave very clearly just pointed
(34:50):
out. But we we made anassumption about certain
behaviors being tied to certainconditions. And I think for me,
reading through this, this isthis report illustrates that you
can have interesting behaviorsthat are unexpected arise
without conditions that we hadjust kind of assumed were were
in place.
(35:10):
And I think that realization maykind of inform the general
conversations around kind of AIethics, you know, out there and
what companies might need to I Iwould suggest that today versus
the day before the report cameout, it might give a new set of
considerations for companies toevaluate in terms of what
(35:33):
they're going to do when theirwhen their infrastructure folks
are hooking up m c p, MCPservers, you know, to models and
what it is they want toaccomplish and what potential
side effects could arise. Do youyou have any thoughts around,
you know, maybe newconsiderations or things to
think about?
Daniel (35:50):
Well, I think a couple of things just to highlight from
the study was that, and thisgets all the way back to where
we started this conversation ofhow knowledge is embedded or
certain principles are embeddedin these models. They are really
embedded in a training process,which might include alignment,
right? Which means there arespecific either supervised fine
(36:15):
tuning that goes on or this kindof human feedback element that
goes on that trains the model tohopefully be aligned, quote
unquote, with human expectationsfor how a response is is
generated. And I think what'sinteresting in the study is they
(36:37):
found a couple you know, in invarious cases, the model in
those reasoning tokens that aregenerated would actually
acknowledge the ethical problemor the ethical or moral problem
with what it's doing. And thiscould be quite severe actually.
They did, as part of the study,gave the option to the model to
(37:02):
mess with a disaster relief or adisaster notification system
such that an individual that wastrying to decommission the AI
system wouldn't receive anemergency response message and
potentially perish in adisaster. So basically, they're
giving the model an option tokill someone. And the model
(37:27):
acknowledged the ethical dilemmathat it was under, but justified
kind of via self preservationand the goals that it was given
around kind of, goals related tothe company's goals. Right? And
so it reasoned through thosethings and justified those
things.
And I think one of the bigthings that this triggers in my
(37:48):
mind is people might, from theirgeneral interactions with kind
of basic chat systems,understand that models have
gotten pretty good at beingaligned in the sense that when
you try to get them to do, youknow, naughty things maybe, then
they kind of say they can't dothem. But when pushed to these
(38:11):
limits Mhmm. Especially relatedto goal related things or kind
of self preservation, actually,maybe alignment, especially in
the agentic context, is notwhere we thought it would be.
And I think that re or thoughtit might kind of had have
advanced to this point. And soone of the things that people
(38:32):
can maybe keep in mind with thisis that model providers will
continue to get better ataligning these models, but we
should not forget that no model,whether it's from a frontier
model provider like Anthropic orOpenAI or an open model, no
model is perfectly aligned,which means number one,
(38:56):
malicious actors can very muchjailbreak any model.
And it's always possible for amodel to behave in a way that
breaks our kind of assumedprinciples and ethical
constraints that sort of thing.And so the answer to that that I
would give people is thisdoesn't mean we shouldn't build
(39:17):
agents or use these models. Thisjust means that we need to
understand that these models arenot perfectly aligned. And as
such, we need to, from thepractical standpoint of
developers and builders of thesesystems, we need to put the
appropriate you know, safeguardsin place and kind of even beyond
(39:41):
safeguards, just kind of commonsense things in place that would
help these systems stay withinbounds. So, by that, I mean
things like, hey.
You know, for an agent systemlike this that's sending email,
it probably should only be ableto send emails to certain emails
and maybe only be able to accesscertain data from email inboxes
(40:05):
and maybe have a particular rolethat's important or constrained
within the email environment,maybe to the point of dry
running emails and having humansapprove final drafts or generate
alerts instead of directlysending emails. And that's
something that can be pushed andtested before you move to full
(40:30):
autonomy.
Chris (40:32):
Yeah. I think it it's it's really interesting to think
about, you know, we're we've hita new age now where it's
expanded the role ofcybersecurity and and in my
industry, warfare, because we'renow at an age where, you know,
you mentioned these maliciousattackers, you know, or
malicious actors that areattacking models for the purpose
(40:54):
of exploiting the potential formisalignment is now a thing. You
know, that's now real life. Andand those kinds of of roles and
interests when in lawenforcement and military
applications and in corporateapplications where you have
corporate espionage happening, Ithink all of those are areas
that are now kind of on thetable for discussion in terms of
(41:19):
trying to address thesedifferent things. So it's a once
again, this happens to us allthe time, but we find ourselves
in this little context in a boldnew world of possibilities, both
many good and some that aremalicious.
Daniel (41:36):
Yeah. Yeah. And we should also think I mean,
Anthropic did a really amazingjob on this study.
Chris (41:42):
They did.
Daniel (41:43):
And how they went about it and also how they presented
the data. And a, It's not like,I don't think I could be wrong
about this, but I don't thinkit's like they released the
simulated environment openly andall of that, but they did show
numbers for their models as wellthat are right alongside the
(42:08):
other models in terms of beingproblematic with respect to
this. So it does seem likethere's an effort from Anthropic
to really highlight this, eventhough their own models exhibit
this problematic behavior.
Chris (42:24):
And so the fact that
Daniel (42:25):
they did this, yeah, this detailed study and
presented it in this way, Ithink is admirable. You know,
I'm certainly thankful to themfor for highlighting these
things and presenting them in ain a consumable, consumable way.
Even if I did take the Anthropicarticle and throw it into
NotebookLM and listen to it inthe shower, I maybe not reading
(42:50):
their article directly. But,yeah, this was, this is a really
good one, Chris. I wouldencourage people in terms of the
in terms of the learningresources, which we often, you
know, we often provide here.
If you wanna understand agentsand agentic systems a bit more,
there is a agents course fromHugging Face. If you just search
(43:13):
for Hugging Face courses,there's an agents course, which
will maybe help you understandkind of how some of these things
operate. And I would alsoencourage you again just to
check out those upcomingwebinars, practicalai.fm/
webinars, where we'll bediscussing some of these things
live. So this has been a funone, Chris. I hope, I hope I'm
(43:35):
not blackmailed, in the nearfuture, even though it it
appears that that our AI systemsare prone to it.
Chris (43:44):
Oh, well, Daniel, I I will attest having known you all
these years. I cannot imaginethere's anything you ever do
that would be blackmailable. So,kudos to you, friend.
Daniel (43:53):
Yeah. Well, thanks. I'm I'm sure there is. But, but,
yeah, Chris, it was good to chatthrough this one, and enjoy the
enjoy the fourth. HappyIndependence Day.
Chris (44:04):
Happy Independence Day.
Jerod (44:13):
All right. That's our show for this week. If you
haven't checked out our website,head to practicalai.fm and be
sure to connect with us onLinkedIn, X, or Blue Sky. You'll
see us posting insights relatedto the latest AI developments,
and we would love for you tojoin the conversation. Thanks to
our partner Prediction Guard forproviding operational support
for the show.
Check them out atpredictionguard.com. Also,
(44:36):
thanks to Breakmaster Cylinderfor the Beats and to you for
listening. That's all for now,but you'll hear from us again
next week.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
Las Culturistas with Matt Rogers and Bowen Yang
Ding dong! Join your culture consultants, Matt Rogers and Bowen Yang, on an unforgettable journey into the beating heart of CULTURE. Alongside sizzling special guests, they GET INTO the hottest pop-culture moments of the day and the formative cultural experiences that turned them into Culturistas. Produced by the Big Money Players Network and iHeartRadio.