Humans Need Entropy

How humans and AI models both share the weakness of deterioration without novel inputs. 

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Episode Transcript

S1 (00:00):
I've had several thoughts on the Carpathian Dwarkesh conversation that
took place in late October of 25, but the one
that keeps haunting me is something Karpathy just kind of
casually mentioned before moving on to another topic. I think
it might be the biggest idea in the whole conversation.
He was talking about human model similarities and he says

(00:23):
humans collapse during the course of their lives. Children have
an overfit, yet they will say stuff that will shock
you because they're not yet collapsed. But we adults, we
end up revisiting the same thoughts. We end up saying
more and more the same stuff. The learning rate goes down,

(00:43):
the collapse continues to get worse, and then everything deteriorates.
End quote. Since my 20s, I've been terrified of this
happening to me. It pierces my soul whenever my partner
says things like, I knew you were going to say that. Ouch.

(01:04):
Predictable humor or wit isn't another example. How many older
people do you know who tell the same stories and
jokes over and over? They watch the same shows. They
listen to the same five bands, and then eventually, like
2 or 1, their aperture slowly shrinks until they die. Luckily,

(01:26):
Karpathy gives a solution right after we have to find
sources of entropy. When we were kids, everything was entropy
because everything was new. So we were constantly changing our preferences,
our behaviors, our language and everything. It made us fresh, unpredictable,

(01:47):
which is highly related to the concept I'm obsessed with
from Shannon's theory of information, which in his model defines
information as the part of transmission that isn't repeated or noise.
I think about this constantly when I'm giving talks or
participating in panels or whatever, or when I'm watching someone

(02:07):
else do so. The main thing I'm asking myself, especially
for my own content, is how much of this is new?
How often will I'm presenting this? Will the viewer be
pleasantly surprised? If the answer is not very often I
redo it or I start over. I'm actively doing a
bunch of stuff in addition to pathological reading to maximize

(02:29):
entropy in my life. I'm reading a lot of old
books on writing, like rhetorical figures and stuff like that,
to try to get fresh phrases into my mind. I
regularly reread and listen to Christopher Hitchens books and debates.
Just having exposure to that level of non cliché language.

(02:50):
And I'm currently building in cloud code a skill called
Increase Entropy that incorporates all of this old and fresh
language like a particle accelerator. So I can point it
at a thought or a piece of content and basically
come up with novel ways of saying the same thing.
So I give it the way that I would say

(03:10):
it in a kind of like just breaks me out
of my mold. I even went so far in 2024
to create an AI prompt in fabric that would rate talks, blogs, panels,
or whatever for wows per minute, meaning how often a
given piece of content surprised the audience. I mean, this
was a problem before AI, and now many are delegating

(03:33):
even more and more of their thinking to a system
that learns by crunching mediocrity from the internet. I can
see things getting way worse in this respect. I guess
it's somewhat comforting that this happens to both AI models
and to people. It makes the whole thing more human somehow.

(03:55):
And hearing Karpathy say it so plainly was jarring to
me in a pleasant way. At least for us humans.
The solution seems something like recognize that this is a
problem that starts for everyone in there, probably like mid
to late 20s, and constantly seek and consume sources of
novelty and freshness to maintain young mind.