Curt Tigges (@CurtTigges)

Pinned post

We recently went viral on X with the claim that neurons are "already clean." But what do we mean by that? Are neurons are a useful basis for interp? We're finally releasing our technical report on LLM neurons--and what we've found is far more interesting than mono/polysemanticity.x.com/GoodfireAI/sta…

Goodfire @GoodfireAI 14h

In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety.

Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵

View quoted post on X

Curt Tigges @CurtTigges 6d

very exciting work from some of my colleagues at Goodfire. it turns out we *can* train on interp and make meaningfully positive changes to model behavior, as long as we understand how the model shifts in response (which we definitely need to be careful about!)

Goodfire @GoodfireAI 6d

We cut through the mystery around interpretability in training.

A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.

We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.

But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵

View quoted post on X

Curt Tigges @CurtTigges Oct 22

now is a good time to repost Andrej Karpathy's blog post on digital hygiene: karpathy.bearblog.dev/digital-hygien…

karpathy.bearblog.dev Digital hygiene A guide to basic digital hygiene.

Mckay Wrigley Verified account @mckaywrigley Oct 21

you have a moral responsibility to educate your family & friends on what is going on in cyber rn.

it is only going to get worse/weirder from here.

here’s a great starter todo list to share:
View quoted post on X

Curt Tigges @CurtTigges Oct 21

very excited to see this come out--it provides substantial evidence for the neuron hypothesis. we've replicated the findings in Qwen 3.6 35B-A3B (an MoE) and have some interesting extensions to it in our technical paper!

nev @neverrixx Oct 20

There has been much discussion recently about how interpretable neurons in LLMs are. Today we're announcing our paper from January's cohort of Anthropic Fellows, where we show neurons may be a stronger basis for interpretability than commonly assumed. 🧵 (1/6)

View quoted post on X

Curt Tigges @CurtTigges Oct 16

this is exactly the right takeaway from the neuron work. we're not nearly finished yet; what we have done is found that some of the basic building blocks of what we want to build are more tractable than we thought. but there is so much left to do--and also a way forward to do it!

Eric Ho Verified account @eric_ho Oct 16

nick's post doesn’t mean that we should pack up our bags and that interp and alignment are solved. in fact, i think people should have the opposite reaction.

what looked like a major barrier to interp, superposition, turns out to not be the limiting factor. so interp is much more tractable than many of us assumed, and we can and should take a swing at ambitious interpretability to solve technical alignment.

we also aren't going to one shot interpretability. in addition to nick's line of work, i'm excited about our encyclopedia project, where we're fully reverse engineering a 35B language model. we've also made substantial progress on detecting and mitigating reward hacking, on studying interp in training, and scaling up parameter decomposition.

interp is really starting to work, and this means more people should work on it, not fewer
View quoted post on X

Curt Tigges

Latest posts 5