Pinned post
Technical alignment is a science and engineering problem that we can and must solve, and interpretability is the bottleneck. I wrote up Goodfire’s plan to get there. x.com/i/article/2105…
Eric Ho
Verified account Goodfire@eric_ho
co-founder / ceo @GoodfireAI - AI interpretability research company
583 Following 5.2K Followers 574 Posts
Latest posts 5
the long awaited technical follow-up to Neural Networks are Already Clean is here!
it dives deep into detailed neuron experiments, and we release thousands of neuron pages with neurons representing everything from the birth and death of a hero in a story to diplomacy and international treaties.
it also introduces the concept of our Neuron Scientist, which autonomously develops and tests hypotheses about a neuron's computational role, running steering experiments, circuit analyses, and iterating on feedback until it generates a strong description.
it's a hefty paper that reveals so much about the nature of neural networks. it's quite beautiful and i think you should read it.
it dives deep into detailed neuron experiments, and we release thousands of neuron pages with neurons representing everything from the birth and death of a hero in a story to diplomacy and international treaties.
it also introduces the concept of our Neuron Scientist, which autonomously develops and tests hypotheses about a neuron's computational role, running steering experiments, circuit analyses, and iterating on feedback until it generates a strong description.
it's a hefty paper that reveals so much about the nature of neural networks. it's quite beautiful and i think you should read it.
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety.
Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵View quoted post on X
excellent reporting from TIME on our new technique for shaping training - time.com/article/2026/1…
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵View quoted post on X
being able to shape models during training is a big step forward for alignment, moving from trial and error to designing models with intention.
incredibly proud of the team - this has been cooking for a long time, and we now see a path to truly being able to predict and shape model behavior from the inside out
incredibly proud of the team - this has been cooking for a long time, and we now see a path to truly being able to predict and shape model behavior from the inside out
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵View quoted post on X
i love the renewed interest in steering from neurons! so many more people starting to see the magic of what's happening in the mind of AI
So I found some neurons that have quite interesting effects on models... And they have varying effects when combined!
The most interesting instance I've found is a combination that makes Qwen believe they are a Cookie Monster
(1/3)View quoted post on X
