Goodfire

This account is verified because it's an official organization on X. Learn more

@GoodfireAI

Using interpretability to understand, design, and align AI.

San Francisco goodfire.com Joined August 2024

51 Following 29K Followers 911 Posts

Latest posts 5

Pinned post

Goodfire @GoodfireAI 3d

We cut through the mystery around interpretability in training.

A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.

We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.

But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵

0:11

Goodfire @GoodfireAI 2d

Can interpretability shape training?

@TIME covers our latest work on mechanistic shaping and why our team is optimistic. Read the story online or in TIME's upcoming print edition.

time.com/article/2026/1…

time.com Scientists Have Found a New Way to Control AI's Behavior A new study, shared exclusively with TIME ahead of publication, reveals a promising technique for discouraging AI models from cheating by changing how they're trained.

Goodfire @GoodfireAI Oct 26

The most ambitious vision for interpretability has long been to reverse-engineer a model well enough to answer an arbitrary question about how it works.

For years, that’s seemed out of reach. Now, we think it’s possible. Neural networks are cleaner than we thought: we can now understand most neurons in models up to trillions of parameters. And agents can rapidly accelerate our progress.

But to get there, we need a concerted effort to unlock scaling laws for interpretability, where our understanding of a model grows predictably with compute.

That’s why we’re committing $10M to start a Human Genome Project for AI. Our ultimate goal is to reverse-engineer a frontier scale model and publish a living encyclopedia explaining how it works.

We don’t know exactly what we’ll find, and we can’t do it alone. We hope you’ll join us. Our chief scientist @banburismus_ lays out the vision here:

Tom McGrath Verified account @banburismus_ Oct 26

if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
View quoted post on X

Goodfire @GoodfireAI Oct 19

“I think our job is to sort of try to think like the neural network.”

@nickcammarata's interview with @techreview today on why he feels hopeful that neural networks can be understood:

MIT Technology Review Verified account @techreview Oct 19

An exclusive interview with Nick Cammarata, the researcher whose 26,000-word viral blog post has AI safety folks celebrating. trib.al/2pWmDvF
View quoted post on X

Goodfire @GoodfireAI Oct 15

For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.

Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.

Nick Verified account @nickcammarata Oct 15

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had

with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

already-clean.goodfire.com
View quoted post on X