September 10, 2026 • 4 mins
An artificial intelligence researcher who worked with Anthropic recently quit the industry, and accused the AI giants of 'gambling with our lives' to grow the technology.
Jacob Coxon spent the past three years pre-training AI models, beginning at OpenAI and moving to Anthropic, but has since claimed neither of the big companies are 'acting responsibly'.
Fisher Funds expert Sam Dickie says this is the latest in the case of AI experts 'sounding the alarm' over the risks of the technology.
LISTEN ABOVE
See omnystudio.com/listener for privacy information.
Listen
Watch
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
Speaker 1 (00:00):
Now, as you well know, an anthropic researcher has quit,
saying the industry is gambling with our lives. This comes
just a few weeks after Sam Dickey from Fisher Funds
talked about the AI safety and research company and what
could be the biggest initial public offering in history. And
Sam is back with us.
Speaker 2 (00:14):
Hi, Sam. Good evening, Heather.
Speaker 3 (00:17):
So what has actually happened here?
Speaker 2 (00:20):
Well, this one really went viral, but many high-profile AI
executives have been sounding the alarm saying, And we talked
about Dario himself, who's the founder of Anthropic, when you
and I talked three weeks ago, was talking about the
risks of his product. But Jacob Coxon just resigned from
Anthropic after three years working at both Anthropic and OpenAI.
(00:41):
And he said these leading edge AI companies are racing
to build increasingly powerful systems before they know how to
make them completely safe. And then what the real twist
there was, Evan, who is still employed at Anthropic, publicly
backed him, putting the chance of AI killing all humans
within the next decade or At around 10%, which sounds
really scary, but there's a couple of important qualifications there.
(01:02):
The first one is, he also says the risk from
today's AI is low. Everyone can tap the brakes if
they want, but they're worried the other one won't. So
it's that prisoner's dilemma. So everyone has their foot to
the floor, flat out race.
Speaker 3 (01:17):
Okay, so what is it that they're actually worried is
going to happen?
Speaker 2 (01:21):
Well, it's not a robot suddenly deciding to take over Auckland.
The fear is AI is improving AI. So recursive self-improvement.
So one generation of AI helping build the next and
the next until the machines are improving faster than humans
can keep up. We are not there today. But there
has been some recent incidents that have created a bit
(01:42):
of hysteria, the hugging face hacking incident, for example. And
it's revealing something more mundane, actually, AI models being tested
in setup error so that's not a machine plotting its
escape and it's just humans that got the safety barrier
wrong so what may be more useful as a warning
(02:04):
to us and to to ai model manufacturers is we're
building increasingly powerful technology increasingly quickly and the models become
more capable the blast radius of a simple human mistake
is significantly larger so for us we just need to
be aware of what data you want ai models to access.
Speaker 3 (02:23):
Yeah Now, this is obviously a really bold claim. So
what's the counter-argument?
Speaker 2 (02:27):
Well, the strong counter-argument is, and we talked about this
a couple of weeks ago, is with founder Dario and
many people within Anthropic themselves, they spend an awful amount
of time warning us about their own product. And if
you think about it, it's called regulatory capture, arguably. So
if governments believe AI could be extraordinarily dangerous, they regulate it.
And regulation, of course, is much easier... for a company
(02:51):
with hundreds of billions of dollars to absorb than a
new competitor. So it really shuts out smaller competitors. That's
why it's called regulatory capture. So the fear itself can
become a competitive advantage. But here's the twist. The fear
can be sincere and commercially useful. Those aren't contradictory. So remember,
Anthropic itself built a model called Mythos, and they said
it's too powerful for you guys. But that was a
(03:13):
wonderful marketing campaign as well.
Speaker 3 (03:16):
Now, what does this mean for investors, do you think, Sam?
Speaker 2 (03:20):
I don't think we need to be frightened of our
phone quite yet. But I think the better question is
for all of us is how much freedom should we
give increasingly powerful AI and who gets to decide that?
So if you listen to Dario from Anthropic and some
of the more overzealous Anthropicas, they think they're the only
company on earth that has got the God-given talent and
(03:42):
right to control this beast that they're creating. So for investors,
we should probably watch three things. How fast the technology improves,
particularly that recursive self-improvement where AI is building AI at
a faster pace, how governments respond, because when this fear-mongering
gets viral, you can imagine that governments might do some
silly things, and whether these companies themselves can prove they
(04:04):
can make it safe enough to trust.
Speaker 3 (04:07):
Yeah, very good point. Hey, thank you very much, Sam.
Really appreciate it, as always. We'll talk to you next week.
Sam Dickey, official funds 20 away from seven.
Speaker 2 (04:13):
For more from Heather Duplessy-Allen Drive, listen live to News
Talk ZB from 4 p.m. weekdays or follow the podcast
on iHeartRadio.
Now, as you well know, an anthropic researcher has quit,
saying the industry is gambling with our lives. This comes
just a few weeks after Sam Dickey from Fisher Funds
talked about the AI safety and research company and what
could be the biggest initial public offering in history. And
Sam is back with us.
Speaker 2 (00:14):
Hi, Sam. Good evening, Heather.
Speaker 3 (00:17):
So what has actually happened here?
Speaker 2 (00:20):
Well, this one really went viral, but many high-profile AI
executives have been sounding the alarm saying, And we talked
about Dario himself, who's the founder of Anthropic, when you
and I talked three weeks ago, was talking about the
risks of his product. But Jacob Coxon just resigned from
Anthropic after three years working at both Anthropic and OpenAI.
(00:41):
And he said these leading edge AI companies are racing
to build increasingly powerful systems before they know how to
make them completely safe. And then what the real twist
there was, Evan, who is still employed at Anthropic, publicly
backed him, putting the chance of AI killing all humans
within the next decade or At around 10%, which sounds
really scary, but there's a couple of important qualifications there.
(01:02):
The first one is, he also says the risk from
today's AI is low. Everyone can tap the brakes if
they want, but they're worried the other one won't. So
it's that prisoner's dilemma. So everyone has their foot to
the floor, flat out race.
Speaker 3 (01:17):
Okay, so what is it that they're actually worried is
going to happen?
Speaker 2 (01:21):
Well, it's not a robot suddenly deciding to take over Auckland.
The fear is AI is improving AI. So recursive self-improvement.
So one generation of AI helping build the next and
the next until the machines are improving faster than humans
can keep up. We are not there today. But there
has been some recent incidents that have created a bit
(01:42):
of hysteria, the hugging face hacking incident, for example. And
it's revealing something more mundane, actually, AI models being tested
in setup error so that's not a machine plotting its
escape and it's just humans that got the safety barrier
wrong so what may be more useful as a warning
(02:04):
to us and to to ai model manufacturers is we're
building increasingly powerful technology increasingly quickly and the models become
more capable the blast radius of a simple human mistake
is significantly larger so for us we just need to
be aware of what data you want ai models to access.
Speaker 3 (02:23):
Yeah Now, this is obviously a really bold claim. So
what's the counter-argument?
Speaker 2 (02:27):
Well, the strong counter-argument is, and we talked about this
a couple of weeks ago, is with founder Dario and
many people within Anthropic themselves, they spend an awful amount
of time warning us about their own product. And if
you think about it, it's called regulatory capture, arguably. So
if governments believe AI could be extraordinarily dangerous, they regulate it.
And regulation, of course, is much easier... for a company
(02:51):
with hundreds of billions of dollars to absorb than a
new competitor. So it really shuts out smaller competitors. That's
why it's called regulatory capture. So the fear itself can
become a competitive advantage. But here's the twist. The fear
can be sincere and commercially useful. Those aren't contradictory. So remember,
Anthropic itself built a model called Mythos, and they said
it's too powerful for you guys. But that was a
(03:13):
wonderful marketing campaign as well.
Speaker 3 (03:16):
Now, what does this mean for investors, do you think, Sam?
Speaker 2 (03:20):
I don't think we need to be frightened of our
phone quite yet. But I think the better question is
for all of us is how much freedom should we
give increasingly powerful AI and who gets to decide that?
So if you listen to Dario from Anthropic and some
of the more overzealous Anthropicas, they think they're the only
company on earth that has got the God-given talent and
(03:42):
right to control this beast that they're creating. So for investors,
we should probably watch three things. How fast the technology improves,
particularly that recursive self-improvement where AI is building AI at
a faster pace, how governments respond, because when this fear-mongering
gets viral, you can imagine that governments might do some
silly things, and whether these companies themselves can prove they
(04:04):
can make it safe enough to trust.
Speaker 3 (04:07):
Yeah, very good point. Hey, thank you very much, Sam.
Really appreciate it, as always. We'll talk to you next week.
Sam Dickey, official funds 20 away from seven.
Speaker 2 (04:13):
For more from Heather Duplessy-Allen Drive, listen live to News
Talk ZB from 4 p.m. weekdays or follow the podcast
on iHeartRadio.
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Bobby Bones Show
Listen to 'The Bobby Bones Show' by downloading the daily full replay.