Amil Dravid (@_AmilDravid)

Pinned post

Check out this work co-led by @sophielwang. We find that base models can match much of RL’s reasoning gains with the right first tokens, which we call 𝘵𝘰𝘬𝘦𝘯 𝘤𝘶𝘦𝘴. The effective cue differs across models and can be as simple as “.\n\nOkay,” or “To determine,” with no explicit instruction to reason. We dissect this phenomenon through a set of interventions on RL and the base-model training data itself. We also extend our analysis to a safety case study of refusal and compliance.

Token cues can arise from simple associations in reasoning-related training data, and we can even turn the word “chicken” into a reasoning cue through counterfactual mid-training runs.

Super excited to find a rich underlying phenomenology! Check out Sophie’s thread for the main findings. I’ll share a few other observations below that I found especially cool. 👇

Sophie Wang @SophieLWang Oct 6

Can “chicken” make a base model reason? 🐔

Yes!

Why? With the right first tokens, base models can match RL in reasoning. We trace this to learned training data associations and use a simple data edit to make “chicken” elicit reasoning too.

blog: sophielwang.com/cues

View quoted post on X

Amil Dravid @_AmilDravid Nov 1

I'll be in NYC Nov. 2-3, and Boston Nov. 4-11. Hit me up if you want to meet! Two recent works I'm excited to share:

Neuron universality, superposition, and scaling laws: arxiv.org/abs/2606.03990

Understanding how data shapes RL (w/@SophieLWang):
arxiv.org/abs/2610.06851

Amil Dravid @_AmilDravid Nov 1

I can see where Andrew is coming from. Interpretability has gotten a particular connotation: SAEs, neurons, and an alignment-specific framing, promoted by a few loud Twitter voices. I even sometimes hesitate to say what I work on is “interpretability” because of exactly the reaction Andrew has.

But interpretability is bigger than any one method or agenda. We all need to be open-minded: both the insiders and the critics. After all, we're all in this together: towards a science of deep learning.

Andrew Gordon Wilson Verified account @andrewgwils Oct 31

Mechanistic interpretability was a huge missed opportunity. Many people wanted to join that effort, assuming it meant a mechanistic understanding of generalization. Instead, it was a bait and switch into an extremely niche, esoteric, and largely ill-conceived effort around understanding the roles of individual neurons. This has often been a challenge in alignment and interpretability: compelling initiatives that ought to be broadly defined, or at least mean what we would expect they mean, end up being restricted to a very narrow and esoteric set of topics. For example, if someone says they work on “interpretability” it does not mean what one would naively assume, e.g., interpretable methods, algorithms, representations, or model deconstruction, understanding the data, etc., but rather sparse autoencoders, influence functions, saliency maps. What a waste!
View quoted post on X

Amil Dravid @_AmilDravid Oct 30

Check out our paper led by @nico_dufour on interpreting Neural Thickets from the logit space.

Nicolas DUFOUR @nico_dufour Oct 30

Neural Thickets (@yule_gan @phillip_isola): add random noise to an LLM's weights, keep the best copies, let them vote. Reasoning jumps, no training.
Why does it work? It's sampling in disguise. Swap noise on billions of weights for one random vector on the logits: same gains 🧵

View quoted post on X

Amil Dravid @_AmilDravid Oct 18

If you're interested in advancing our scientific understanding of deep learning, apply to @FlorentinGuth!

Florentin Guth @FlorentinGuth Oct 18

I'm delighted to announce that I'll join @MITEECS as an Assistant Professor next Fall! I will be recruiting graduate students this cycle, please apply if you're interested in the science of deep learning! Despite the ambient doomerism, now is a great time to do research.
View quoted post on X

Amil Dravid

Latest posts 5