TL;DR: We train LLMs to accept LLM neural activations as inputs and answer arbitrary questions about them in natural language. These Activation Oracles generalize far beyond their training distribution, for example uncovering misalignment or secret knowledge introduced via fine-tuning. Activation Oracles can be improved simply by scaling training data quantity and diversity.
The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post.
Thread
New paper:
We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language.
We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so.
We aim to make a general-purpose LLM for explaining activations by:
1. Training on a diverse set of tasks
2. Evaluating on tasks very different from training
This extends prior work (LatentQA) that studied activation verbalization in narrow settings.
Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies.
Applying Activation Oracles is easy. Choose the activation (or set of activations) you want to interpret and ask any question you like!
We [...]
---
Outline:
(00:46) Thread
(04:49) Blog post
(05:27) Introduction
(07:29) Method
(10:15) Activation Oracles generalize to downstream auditing tasks
(13:47) How does Activation Oracle training scale?
(15:01) How do Activation Oracles relate to mechanistic approaches to interpretability?
(19:31) Conclusion
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
December 18th, 2025
Source:
https://www.lesswrong.com/posts/rwoEz3bA9ekxkabc7/activation-oracles-training-and-evaluating-llms-as-general
---
Narrated by TYPE III AUDIO.
---
The below is a reproduction of our X thread on this paper and the Anthropic Alignment blog post.
Thread
New paper:
We train Activation Oracles: LLMs that decode their own neural activations and answer questions about them in natural language.
We find surprising generalization. For instance, our AOs uncover misaligned goals in fine-tuned models, without training to do so.
We aim to make a general-purpose LLM for explaining activations by:
1. Training on a diverse set of tasks
2. Evaluating on tasks very different from training
This extends prior work (LatentQA) that studied activation verbalization in narrow settings.
Our main evaluations are downstream auditing tasks. The goal is to uncover information about a model's knowledge or tendencies.
Applying Activation Oracles is easy. Choose the activation (or set of activations) you want to interpret and ask any question you like!
We [...]
---
Outline:
(00:46) Thread
(04:49) Blog post
(05:27) Introduction
(07:29) Method
(10:15) Activation Oracles generalize to downstream auditing tasks
(13:47) How does Activation Oracle training scale?
(15:01) How do Activation Oracles relate to mechanistic approaches to interpretability?
(19:31) Conclusion
The original text contained 3 footnotes which were omitted from this narration.
---
First published:
December 18th, 2025
Source:
https://www.lesswrong.com/posts/rwoEz3bA9ekxkabc7/activation-oracles-training-and-evaluating-llms-as-general
---
Narrated by TYPE III AUDIO.
---
Mark as Played
Popular Podcasts
Stuff You Should Know
If you've ever wanted to know about champagne, satanism, the Stonewall Uprising, chaos theory, LSD, El Nino, true crime and Rosa Parks, then look no further. Josh and Chuck have you covered.
Dateline NBC
Current and classic episodes, featuring compelling true-crime mysteries, powerful documentaries and in-depth investigations. Follow now to get the latest episodes of Dateline NBC completely free, or subscribe to Dateline Premium for ad-free listening and exclusive bonus content: DatelinePremium.com
The Breakfast Club
The World's Most Dangerous Morning Show, The Breakfast Club, With DJ Envy, Jess Hilarious, And Charlamagne Tha God!