Hadas Orgad

Code for "Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types" is now available!

It includes B-TAP, the interp method behind our work, for localizing and causally intervening on critical weights.

Code + project page + paper 👇🏻

Visiting Paris for @NeurIPSConf? Join us at the ELLIS UnConference!

I'll be sharing my thoughts on the state of interpretability, some open questions, and what I think we should be focusing on.

Unified represenations across modalities? An intriguing work from @benno_krojer and others

Looking for qualified ICLR reviewers for an emergency review on interpretability and safety papers! Please DM 🙏🏻

Since a reporter recently reached out to me about this, and inspired by @ChrisGPotts sharing his own comments publicly, I thought I'd share my perspective here as well.

I find the reassessment of superposition genuinely interesting, especially given how influential the superposition hypothesis has been. In particular, the discussion of how conclusions drawn from toy models may not carry over to real networks with real data is important. Clearly more work is needed to support the broader claims. I'd particularly like to see systematic evaluations of how well the proposed neuron interpretations generalize across inputs and models.

Superposition hasn't played a major role in my own interpretability research. It's one hypothesis in a much broader field, with many different approaches and research questions. As such, these findings don't fundamentally change the questions I'm working on, but they add to a broader view on how models store information.

For me, an important goal of interpretability is to go beyond explanations. Can our understanding help us predict model behavior, improve training methods or make targeted causal interventions? Our B-TAP method, for example, uses interventions analogous to brain lesions to study mechanisms of particular behaviors, without requiring a complete neuron-level explanation.

But most importantly, understanding individual neurons is not the same as understanding complex model behavior. Even if most neurons are interpretable, we still need to understand how they interact to produce behavior. This is especially true if we want to apply interpretability to safety, which includes very complicated behaviors over many tokens and turns such as reward hacking and deception. Trying to apply interpretability to these realistic scenarios is what we should focus on.