July 17, 2025 • 11 mins
In this episode of Hashtag Trending, host Jim Love delves into a series of AI-related stories. Google's Gemini AI withdraws from a chess match against an Atari 2600 chess engine after realizing its limitations. Researchers from MIT and EPFL identify a 'phase transition' in AI language models where they begin to understand semantics over syntax. The episode also highlights the growing issue of AI-generated 'slop,' which overwhelms content reviewers and dilutes quality across various fields. Lastly, Amazon's strategic investment in Anthropic is explored, focusing on infrastructure rather than consumer-friendly AI applications, potentially positioning Amazon as a key player in the AI revolution.
00:00 Introduction and Overview
00:27 Gemini AI vs. Atari Chess Challenge
02:15 AI's Leap to Understanding Meaning
05:38 The Rise of AI-Generated Content
08:49 Amazon's Strategic Investment in AI
11:22 Conclusion and Upcoming Shows
Mark as Played
Transcript
Episode Transcript
Available transcripts are automatically generated. Complete accuracy is not guaranteed.
(00:00):
Gemini backs down froman Atari chess challenge.
Scientists discover the point whereAI makes the leap to understanding
meaning The emergence of AI slopthreatens to drown out good content,
and Amazon may have a unique andprofitable strategy for the internet.
Gold rush.
Welcome to Hashtag Trending on aday full of AI related stories.
(00:24):
I'm your host, Jim Love.
Let's get into it.
Google's Gemini AI officially backedout of a planned chess match against a
46-year-old Atari 2,600 chess engine.
After initially claiming it coulddominate the vintage console, Gemini
reversed course, admitting it wouldstruggle immensely and calling.
(00:46):
cancellation the most timeefficient and sensible decision.
The match was part of a series oflighthearted but revealing tests by Citrix
specialist Robert Caruso, who previouslychallenged ChatGPT and Microsoft
copilot with the same Atari system.
Both of those systems lost badly, andGemini seems to have learned from their
(01:08):
mistakes early on, Gemini confidentlydescribed its ability to evaluate
millions of positions and even suggestedit could beat human chess masters.
but when reminded that the Atari chessengine runs on 128 bytes of Ram and a
1.19 megahertz chip and still outplayedits AI competitors, Gemini reconsidered.
(01:32):
After reevaluating the match, itacknowledged it had been hallucinating
about its own capabilities.
Now, this isn't really about chess.
It highlights the difference betweenlarge language models like Gemini
and Purpose-built chess engines.
LLMs are trained on language, notlogic, trees or game strategy, and
(01:53):
there's another lesson in this.
We tend to think of AI as the answerfor everything, but at least for now,
there appears to be times when a rulesbased algorithm is faster, more accurate,
more reliable, and above all cheaper.
Or as one AI observer puts it,sometimes the smartest move
is knowing when not to play.
(02:15):
Researchers think they've foundthe precise moment when a language
model starts truly understanding.
the meaning of words and sentences.
It was a joint team from MIT and EPFL,the Swiss Federal Institute of Technology
in Lausanne, and it discovered thatas transformer models are trained
(02:36):
with more and more data and more andmore connections built up in their
neural networks, at some point theygo through a sudden measurable shift.
Early in the training, themodels rely heavily on structure.
Things like syntax, word, order,position, and sentence structure.
But after being exposed toenough text, they abruptly start
(03:00):
appearing to understand themeanings of words themselves.
Switching from syntax, themechanical structure and position
to semantics the actual meaning.
It appears that this isnot a gradual evolution.
It is a sharp transition and asudden change in state compared
by the researchers to somethinglike where water, after raising
(03:21):
its temperature suddenly reaches aboiling point and turns into steam.
They call this a phasetransition, and they say it
happens in physics all the time.
In AI models, it appears that a similarthing occurs in the training process.
The model builds internalrepresentations called embeddings.
(03:43):
At first, these embeddings reflect mostlywhere the words appear in sentences
and the model gets better and better atpredicting by word position and structure.
But once the training data passesthat certain threshold, the model
re encodes the embeddings based onhow words relate to each other's
semantically, not their position.
They're meaning, and you can spotthis because the model starts
(04:07):
grouping words with similar meaningsthat may be far apart, and it puts
them closer together in the model.
For example, Cat and Dog might becomerelated in the model's internal map,
regardless of their position in the text.
The breakthrough published in the Journalof Statistical Mechanics might lead
(04:27):
to big changes in how large languagemodels are designed and trained.
AI researchers know that increasingthe amount of data has had a
positive impact on the intelligenceand utility of the model.
Part of this experiment will explainthat, but if we extend the parallel
to a physical phase shift, is there apoint at which adding additional data
(04:51):
has lower or perhaps no additionalimpact on the model's abilities?
Is there a point at whichit will hold the model back?
It suggests that exposing a model tojust enough data to reach this phase
change may be more efficient than simplythrowing more and more data at it.. but
it also gives researchers a start atone of the great questions of our time.
(05:15):
AI researchers will tell you,we really don't understand what
is happening in these models.
And with this theory, we mightbe on our way to, for want of a
better phrase, understanding howAI begins to get language, not just
at the output layer, but deep inthe structure of how it thinks.
(05:38):
We all know that AI can create contentfaster than anything we've ever seen
before, but creating it is only one step.
Somebody has to review it, and acrossall industries, the people responsible
for that work are drowning, inacademia, researchers are overwhelmed.
A recent report in the Guardian showsthat the number of scientific papers
published has exploded in the last decade,
(06:01):
In academia, researchers are overwhelmed.
A recent report in the Guardian shows thatthe number of scientific papers published
has exploded in the last decade, butmany of those papers are of low quality
and some are clearly generated by ai.
One AI written paper about a rat withan anatomically impossible feature
(06:22):
was accepted, published, and onlypulled after readers complained.
The peer review system, which relieson experts volunteering their time
simply can't keep up in cybersecurity.
It's the same pattern.
Organizations run bug bounty programsoffering cash to people who find
security flaws in their code.
(06:45):
But Daniel Stenberg, the creatorof the open source tool, curl said
he's being buried in low effort.
Bug reports, many clearly written by ai.
The reports are often vague, misleading,sometimes just wrong, and each one has
to be reviewed, and that takes timeaway from fixing real vulnerabilities.
(07:08):
In publishing the amount of bookspublished by Amazon's Kindle Direct
Publishing is more than doubled in thepast five years, Amazon is struggling
to cope with a tsunami of low qualitycontent created by AI and that they feel
might be drowning out the great content.
This is a pattern that's beingrepeated over and over and not
(07:29):
just in text-based content.
Some are complaining that YouTube is beingflooded with low quality AI developed
content, Particularly since video andpodcasts became so easy to generate
with ai, some worry that the internetitself may collapse under the weight
of low quality AI generated content.
(07:49):
There's even a term for it.
It's called AI slop.
The message is clear.
AI makes it easy to quickly generateunlimited amounts of content, but it
takes real skill and human effort tomake that content useful and valuable.
When the floodgates open, the overwhelmingwave of not even bad content, but just
(08:12):
content that's not very good can crowdout the excellent and waste the time of
the people we rely on to keep systemstrustworthy and secure, and the content
valuable and it risks turning offpeople who are looking for exciting,
entertaining, and informative content.
(08:32):
In the Klondike Gold Rush, more than100,000 people set out to strike it.
Rich most never made it allthe way to the Klondike.
Only a tiny fraction ofthose found any gold at all.
But you know who really profited?
The ones selling picks andshovels, and Amazon seems to be
(08:55):
taking the same approach with AI.
Reports say that the company's preparing asecond $4 billion investment in Anthropic,
bringing its total stake in the cloudAI developer to more than $8 billion.
But unlike Microsoft and Google,Amazon isn't really pushing a flashy
consumer chatbot or talking about a GI.
(09:17):
It's selling the infrastructurethat runs that ai.
Anthropic trains its models on AmazonWeb Services using Amazon's own
Trainium and Inferentia chips, andthat's already turning into serious
revenue, especially with anthropics.
Focus on enabling business processeswith its specialized models for
(09:39):
specific verticals and segments.
Business Insider reports that Anthropiccould bring in over a billion dollars
in AWS revenue this year with morethan 5 billion annually by 2027.
And Anthropic itself sharesa similar philosophy.
Instead of being all things to allpeople, it's focused on finding areas
(10:00):
where its AI can have the biggest impactin terms of automating and transforming
specific business processes and verticals.
This week it announced it'll beproviding a new offering aimed
solidly at the investment industry.
One that is eagerly anticipatedand which has huge opportunities
in terms of creating and managingresearch and support services.
(10:22):
So by backing Anthropic, Amazon isn'tjust investing in a startup, it's
acquiring a tool that is a showcase forAWS and quietly positioning itself as the
company that powers the AI revolution,regardless of which model wins.
And in doing that, although it'sinvesting billions of dollars,
most of that investment is inthe form of processing cycles.
(10:43):
Largely on its own chips and inthe data centers it needs to build
anyway to support the AI drivengrowth in its hosting business.
It isn't sexy, but neither was Amazon'sfirst strategy of building capacity
early when cloud computing was inthe initial stages, a move that ran
counter to what everyone else wasdoing and thought of as very risky.
(11:07):
But that strategy made themthe biggest hosting company in
the world, both in market shareand revenue and profitability.
So they might be on the vergeof pulling this off for the
second time after the gold rush.
And that's our show.
I try not to do a full programon one topic, but there were just
(11:27):
so many AI stories that deservedattention over the past few days.
I felt I had to run with them.
If you were particularly interested intoday's stories, you might want to tune
in on the weekend when we do our projectSynapse show, where we'll be doing a
deeper dive into these stories and more.
I'm your host, Jim Love.
Have a thrilling Thursday.
Gemini backs down froman Atari chess challenge.
Scientists discover the point whereAI makes the leap to understanding
meaning The emergence of AI slopthreatens to drown out good content,
and Amazon may have a unique andprofitable strategy for the internet.
Gold rush.
Welcome to Hashtag Trending on aday full of AI related stories.
(00:24):
I'm your host, Jim Love.
Let's get into it.
Google's Gemini AI officially backedout of a planned chess match against a
46-year-old Atari 2,600 chess engine.
After initially claiming it coulddominate the vintage console, Gemini
reversed course, admitting it wouldstruggle immensely and calling.
(00:46):
cancellation the most timeefficient and sensible decision.
The match was part of a series oflighthearted but revealing tests by Citrix
specialist Robert Caruso, who previouslychallenged ChatGPT and Microsoft
copilot with the same Atari system.
Both of those systems lost badly, andGemini seems to have learned from their
(01:08):
mistakes early on, Gemini confidentlydescribed its ability to evaluate
millions of positions and even suggestedit could beat human chess masters.
but when reminded that the Atari chessengine runs on 128 bytes of Ram and a
1.19 megahertz chip and still outplayedits AI competitors, Gemini reconsidered.
(01:32):
After reevaluating the match, itacknowledged it had been hallucinating
about its own capabilities.
Now, this isn't really about chess.
It highlights the difference betweenlarge language models like Gemini
and Purpose-built chess engines.
LLMs are trained on language, notlogic, trees or game strategy, and
(01:53):
there's another lesson in this.
We tend to think of AI as the answerfor everything, but at least for now,
there appears to be times when a rulesbased algorithm is faster, more accurate,
more reliable, and above all cheaper.
Or as one AI observer puts it,sometimes the smartest move
is knowing when not to play.
(02:15):
Researchers think they've foundthe precise moment when a language
model starts truly understanding.
the meaning of words and sentences.
It was a joint team from MIT and EPFL,the Swiss Federal Institute of Technology
in Lausanne, and it discovered thatas transformer models are trained
(02:36):
with more and more data and more andmore connections built up in their
neural networks, at some point theygo through a sudden measurable shift.
Early in the training, themodels rely heavily on structure.
Things like syntax, word, order,position, and sentence structure.
But after being exposed toenough text, they abruptly start
(03:00):
appearing to understand themeanings of words themselves.
Switching from syntax, themechanical structure and position
to semantics the actual meaning.
It appears that this isnot a gradual evolution.
It is a sharp transition and asudden change in state compared
by the researchers to somethinglike where water, after raising
(03:21):
its temperature suddenly reaches aboiling point and turns into steam.
They call this a phasetransition, and they say it
happens in physics all the time.
In AI models, it appears that a similarthing occurs in the training process.
The model builds internalrepresentations called embeddings.
(03:43):
At first, these embeddings reflect mostlywhere the words appear in sentences
and the model gets better and better atpredicting by word position and structure.
But once the training data passesthat certain threshold, the model
re encodes the embeddings based onhow words relate to each other's
semantically, not their position.
They're meaning, and you can spotthis because the model starts
(04:07):
grouping words with similar meaningsthat may be far apart, and it puts
them closer together in the model.
For example, Cat and Dog might becomerelated in the model's internal map,
regardless of their position in the text.
The breakthrough published in the Journalof Statistical Mechanics might lead
(04:27):
to big changes in how large languagemodels are designed and trained.
AI researchers know that increasingthe amount of data has had a
positive impact on the intelligenceand utility of the model.
Part of this experiment will explainthat, but if we extend the parallel
to a physical phase shift, is there apoint at which adding additional data
(04:51):
has lower or perhaps no additionalimpact on the model's abilities?
Is there a point at whichit will hold the model back?
It suggests that exposing a model tojust enough data to reach this phase
change may be more efficient than simplythrowing more and more data at it.. but
it also gives researchers a start atone of the great questions of our time.
(05:15):
AI researchers will tell you,we really don't understand what
is happening in these models.
And with this theory, we mightbe on our way to, for want of a
better phrase, understanding howAI begins to get language, not just
at the output layer, but deep inthe structure of how it thinks.
(05:38):
We all know that AI can create contentfaster than anything we've ever seen
before, but creating it is only one step.
Somebody has to review it, and acrossall industries, the people responsible
for that work are drowning, inacademia, researchers are overwhelmed.
A recent report in the Guardian showsthat the number of scientific papers
published has exploded in the last decade,
(06:01):
In academia, researchers are overwhelmed.
A recent report in the Guardian shows thatthe number of scientific papers published
has exploded in the last decade, butmany of those papers are of low quality
and some are clearly generated by ai.
One AI written paper about a rat withan anatomically impossible feature
(06:22):
was accepted, published, and onlypulled after readers complained.
The peer review system, which relieson experts volunteering their time
simply can't keep up in cybersecurity.
It's the same pattern.
Organizations run bug bounty programsoffering cash to people who find
security flaws in their code.
(06:45):
But Daniel Stenberg, the creatorof the open source tool, curl said
he's being buried in low effort.
Bug reports, many clearly written by ai.
The reports are often vague, misleading,sometimes just wrong, and each one has
to be reviewed, and that takes timeaway from fixing real vulnerabilities.
(07:08):
In publishing the amount of bookspublished by Amazon's Kindle Direct
Publishing is more than doubled in thepast five years, Amazon is struggling
to cope with a tsunami of low qualitycontent created by AI and that they feel
might be drowning out the great content.
This is a pattern that's beingrepeated over and over and not
(07:29):
just in text-based content.
Some are complaining that YouTube is beingflooded with low quality AI developed
content, Particularly since video andpodcasts became so easy to generate
with ai, some worry that the internetitself may collapse under the weight
of low quality AI generated content.
(07:49):
There's even a term for it.
It's called AI slop.
The message is clear.
AI makes it easy to quickly generateunlimited amounts of content, but it
takes real skill and human effort tomake that content useful and valuable.
When the floodgates open, the overwhelmingwave of not even bad content, but just
(08:12):
content that's not very good can crowdout the excellent and waste the time of
the people we rely on to keep systemstrustworthy and secure, and the content
valuable and it risks turning offpeople who are looking for exciting,
entertaining, and informative content.
(08:32):
In the Klondike Gold Rush, more than100,000 people set out to strike it.
Rich most never made it allthe way to the Klondike.
Only a tiny fraction ofthose found any gold at all.
But you know who really profited?
The ones selling picks andshovels, and Amazon seems to be
(08:55):
taking the same approach with AI.
Reports say that the company's preparing asecond $4 billion investment in Anthropic,
bringing its total stake in the cloudAI developer to more than $8 billion.
But unlike Microsoft and Google,Amazon isn't really pushing a flashy
consumer chatbot or talking about a GI.
(09:17):
It's selling the infrastructurethat runs that ai.
Anthropic trains its models on AmazonWeb Services using Amazon's own
Trainium and Inferentia chips, andthat's already turning into serious
revenue, especially with anthropics.
Focus on enabling business processeswith its specialized models for
(09:39):
specific verticals and segments.
Business Insider reports that Anthropiccould bring in over a billion dollars
in AWS revenue this year with morethan 5 billion annually by 2027.
And Anthropic itself sharesa similar philosophy.
Instead of being all things to allpeople, it's focused on finding areas
(10:00):
where its AI can have the biggest impactin terms of automating and transforming
specific business processes and verticals.
This week it announced it'll beproviding a new offering aimed
solidly at the investment industry.
One that is eagerly anticipatedand which has huge opportunities
in terms of creating and managingresearch and support services.
(10:22):
So by backing Anthropic, Amazon isn'tjust investing in a startup, it's
acquiring a tool that is a showcase forAWS and quietly positioning itself as the
company that powers the AI revolution,regardless of which model wins.
And in doing that, although it'sinvesting billions of dollars,
most of that investment is inthe form of processing cycles.
(10:43):
Largely on its own chips and inthe data centers it needs to build
anyway to support the AI drivengrowth in its hosting business.
It isn't sexy, but neither was Amazon'sfirst strategy of building capacity
early when cloud computing was inthe initial stages, a move that ran
counter to what everyone else wasdoing and thought of as very risky.
(11:07):
But that strategy made themthe biggest hosting company in
the world, both in market shareand revenue and profitability.
So they might be on the vergeof pulling this off for the
second time after the gold rush.
And that's our show.
I try not to do a full programon one topic, but there were just
(11:27):
so many AI stories that deservedattention over the past few days.
I felt I had to run with them.
If you were particularly interested intoday's stories, you might want to tune
in on the weekend when we do our projectSynapse show, where we'll be doing a
deeper dive into these stories and more.
I'm your host, Jim Love.
Have a thrilling Thursday.
Popular Podcasts
Crime Junkie
Does hearing about a true crime case always leave you scouring the internet for the truth behind the story? Dive into your next mystery with Crime Junkie. Every Monday, join your host Ashley Flowers as she unravels all the details of infamous and underreported true crime cases with her best friend Brit Prawat. From cold cases to missing persons and heroes in our community who seek justice, Crime Junkie is your destination for theories and stories you won’t hear anywhere else. Whether you're a seasoned true crime enthusiast or new to the genre, you'll find yourself on the edge of your seat awaiting a new episode every Monday. If you can never get enough true crime... Congratulations, you’ve found your people. Follow to join a community of Crime Junkies! Crime Junkie is presented by audiochuck Media Company.
24/7 News: The Latest
The latest news in 4 minutes updated every hour, every day.
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.