LessWrong (Curated & Popular)
Audio narrations of LessWrong posts. Includes all curated posts and all posts with 125+ karma. If you'd like more, subscribe to the “Lesswrong (30+ karma)” feed.
Episodes
May 16, 2025 • 2 mins
Epistemic status: thing people have told me that seems right. Also primarily relevant to US audiences. Also I am speaking in my personal capacity and not representing any employer, present or past.
Sometimes, I talk to people who work in the AI governance space. One thing that multiple people have told me, which I found surprising, is that there is apparently a real problem where people accidentally rule themselves out o...
Sometimes, I talk to people who work in the AI governance space. One thing that multiple people have told me, which I found surprising, is that there is apparently a real problem where people accidentally rule themselves out o...
Mark as Played
May 16, 2025 • 5 mins
"If you kiss your child, or your wife, say that you only kiss things which are human, and thus you will not be disturbed if either of them dies." - Epictetus
"Whatever suffering arises, all arises due to attachment; with the cessation of attachment, there is the cessation of suffering." - Pali canon
"He is not disturbed by loss, he does not delight in gain; he is not disturbed by blame,...
"Whatever suffering arises, all arises due to attachment; with the cessation of attachment, there is the cessation of suffering." - Pali canon
"He is not disturbed by loss, he does not delight in gain; he is not disturbed by blame,...
Mark as Played
May 15, 2025 • 8 mins
The other day I discussed how high monitoring costs can explain the emergence of “aristocratic” systems of governance:
Aristocracy and Hostage Capital
Arjun Panickssery · Jan 8
There's a conventional narrative by which the pre-20th century aristocracy was the "old corruption" where civil and military positions were distributed inefficiently due to nepotism until the system was replaced by a ...
Aristocracy and Hostage Capital
Arjun Panickssery · Jan 8
There's a conventional narrative by which the pre-20th century aristocracy was the "old corruption" where civil and military positions were distributed inefficiently due to nepotism until the system was replaced by a ...
Mark as Played
May 14, 2025 • 6 mins
Eliezer and I wrote a book. It's titled If Anyone Builds It, Everyone Dies. Unlike a lot of other writing either of us have done, it's being professionally published. It's hitting shelves on September 16th.
It's a concise (~60k word) book aimed at a broad audience. It's been well-received by people who received advance copies, with some endorsements including:
The most important book I&...
It's a concise (~60k word) book aimed at a broad audience. It's been well-received by people who received advance copies, with some endorsements including:
The most important book I&...
Mark as Played
May 14, 2025 • 8 mins
It was a cold and cloudy San Francisco Sunday. My wife and I were having lunch with friends at a Korean cafe.
My phone buzzed with a text. It said my mom was in the hospital.
I called to find out more. She had a fever, some pain, and had fainted. The situation was serious, but stable.
Monday was a normal day. No news was good news, right?
Tuesday she had seizures.
Wednesday she was...
My phone buzzed with a text. It said my mom was in the hospital.
I called to find out more. She had a fever, some pain, and had fainted. The situation was serious, but stable.
Monday was a normal day. No news was good news, right?
Tuesday she had seizures.
Wednesday she was...
Mark as Played
May 13, 2025 • 4 mins
At the bottom of the LessWrong post editor, if you have at least 100 global karma, you may have noticed this button.
The button Many people click the button, and are jumpscared when it starts an Intercom chat with a professional editor (me), asking what sort of feedback they'd like.
So, that's what it does. It's a summon Justis button.
Why summon Justis?
To get feedback on your...
The button Many people click the button, and are jumpscared when it starts an Intercom chat with a professional editor (me), asking what sort of feedback they'd like.
So, that's what it does. It's a summon Justis button.
Why summon Justis?
To get feedback on your...
Mark as Played
May 8, 2025 • 8 mins
For months, I had the feeling: something is wrong. Some core part of myself had gone missing.
I had words and ideas cached, which pointed back to the missing part.
There was the story of Benjamin Jesty, a dairy farmer who vaccinated his family against smallpox in 1774 - 20 years before the vaccination technique was popularized, and the same year King Louis XV of France died of the disease.
There was a...
I had words and ideas cached, which pointed back to the missing part.
There was the story of Benjamin Jesty, a dairy farmer who vaccinated his family against smallpox in 1774 - 20 years before the vaccination technique was popularized, and the same year King Louis XV of France died of the disease.
There was a...
Mark as Played
May 4, 2025 • 13 mins
(Disclaimer: Post written in a personal capacity. These are personal hot takes and do not in any way represent my employer's views.)
TL;DR: I do not think we will produce high reliability methods to evaluate or monitor the safety of superintelligent systems via current research paradigms, with interpretability or otherwise. Interpretability seems a valuable tool here and remains worth investing in, as it will hopefu...
TL;DR: I do not think we will produce high reliability methods to evaluate or monitor the safety of superintelligent systems via current research paradigms, with interpretability or otherwise. Interpretability seems a valuable tool here and remains worth investing in, as it will hopefu...
Mark as Played
May 3, 2025 • 11 mins
It'll take until ~2050 to repeat the level of scaling that pretraining compute is experiencing this decade, as increasing funding can't sustain the current pace beyond ~2029 if AI doesn't deliver a transformative commercial success by then. Natural text data will also run out around that time, and there are signs that current methods of reasoning training might be mostly eliciting capabilities from the base model.
Mark as Played
In this blog post, we analyse how the recent AI 2027 forecast by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, and Romeo Dean has been discussed across Chinese language platforms. We present:
- Our research methodology and synthesis of key findings across media artefacts
- A proposal for how censorship patterns may provide signal for the Chinese government's thinking about AGI and the race to super...
Mark as Played
April 25, 2025 • 1 min
This is a link post. to follow up my philantropic pledge from 2020, i've updated my philanthropy page with the 2024 results.
in 2024 my donations funded $51M worth of endpoint grants (plus $2.0M in admin overhead and philanthropic software development). this comfortably exceeded my 2024 commitment of $42M (20k times $2100.00 — the minimum price of ETH in 2024).
this also concludes my 5-year donation pledge,...
in 2024 my donations funded $51M worth of endpoint grants (plus $2.0M in admin overhead and philanthropic software development). this comfortably exceeded my 2024 commitment of $42M (20k times $2100.00 — the minimum price of ETH in 2024).
this also concludes my 5-year donation pledge,...
Mark as Played
April 24, 2025 • 15 mins
I’ve been thinking recently about what sets apart the people who’ve done the best work at Anthropic.
You might think that the main thing that makes people really effective at research or engineering is technical ability, and among the general population that's true. Among people hired at Anthropic, though, we’ve restricted the range by screening for extremely high-percentile technical ability, so the remaining diffe...
You might think that the main thing that makes people really effective at research or engineering is technical ability, and among the general population that's true. Among people hired at Anthropic, though, we’ve restricted the range by screening for extremely high-percentile technical ability, so the remaining diffe...
Mark as Played
This is a link post. Guillaume Blanc has a piece in Works in Progress (I assume based on his paper) about how France's fertility declined earlier than in other European countries, and how its power waned as its relative population declined starting in the 18th century. In 1700, France had 20% of Europe's population (4% of the whole world population). Kissinger writes in Diplomacy with respect to the Versailles Peace Confe...
Mark as Played
We’ve written a new report on the threat of AI-enabled coups.
I think this is a very serious risk – comparable in importance to AI takeover but much more neglected.
In fact, AI-enabled coups and AI takeover have pretty similar threat models. To see this, here's a very basic threat model for AI takeover:
I think this is a very serious risk – comparable in importance to AI takeover but much more neglected.
In fact, AI-enabled coups and AI takeover have pretty similar threat models. To see this, here's a very basic threat model for AI takeover:
Mark as Played
April 22, 2025 • 28 mins
Back in the 1990s, ground squirrels were briefly fashionable pets, but their popularity came to an abrupt end after an incident at Schiphol Airport on the outskirts of Amsterdam. In April 1999, a cargo of 440 of the rodents arrived on a KLM flight from Beijing, without the necessary import papers. Because of this, they could not be forwarded on to the customer in Athens. But nobody was able to correct the error and send them back ...
Mark as Played
April 21, 2025 • 10 mins
Subtitle: Bad for loss of control risks, bad for concentration of power risks
I’ve had this sitting in my drafts for the last year. I wish I’d been able to release it sooner, but on the bright side, it’ll make a lot more sense to people who have already read AI 2027.
I’ve had this sitting in my drafts for the last year. I wish I’d been able to release it sooner, but on the bright side, it’ll make a lot more sense to people who have already read AI 2027.
- There's a good chance that AGI will be trained before this decade is out.
- By AGI I mean “An AI system at least as good as the best ...
Mark as Played
April 20, 2025 • 1 min
Though, given my doomerism, I think the natsec framing of the AGI race is likely wrongheaded, let me accept the Dario/Leopold/Altman frame that AGI will be aligned to the national interest of a great power. These people seem to take as an axiom that a USG AGI will be better in some way than CCP AGI. Has anyone written justification for this assumption?
I am neither an American citizen nor a Chinese citizen.
Wh...
I am neither an American citizen nor a Chinese citizen.
Wh...
Mark as Played
Introduction
Writing this post puts me in a weird epistemic position. I simultaneously believe that:- The reasoning failures that I'll discuss are strong evidence that current LLM- or, more generally, transformer-based approaches won't get us AGI
- As soon as major AI labs read about the specific reasoning failures described here, they might fix them
- But future versions...
Mark as Played
Dario Amodei, CEO of Anthropic, recently worried about a world where only 30% of jobs become automated, leading to class tensions between the automated and non-automated. Instead, he predicts that nearly all jobs will be automated simultaneously, putting everyone "in the same boat." However, based on my experience spanning AI research (including first author papers at COLM / NeurIPS and attending MATS under Neel Nanda), ...
Mark as Played
Audio note: this article contains 31 uses of latex notation, so the narration may be difficult to follow. There's a link to the original text in the episode description.
Lewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda
* = equal contribution
The following piece is a list of snippets about research from the GDM mechanistic inter...
Lewis Smith*, Sen Rajamanoharan*, Arthur Conmy, Callum McDougall, Janos Kramar, Tom Lieberum, Rohin Shah, Neel Nanda
* = equal contribution
The following piece is a list of snippets about research from the GDM mechanistic inter...
Mark as Played