@jacobandreas
Pinned post
Higher benchmark scores do not always mean better models for users.
Why? We claim that RL teaches LMs to be correct but not how to be correct: code can pass tests but be unreadable; explanations can be right but unclear.
How do we train LMs to be right in the right way?
(1/n)View quoted post on X
Pluralistic alignment is thriving as a research agenda yet failing at its goal: making the AI systems people actually use more pluralistic馃寛
馃毃New position paper: we argue adoption in deployed models should be the fields main goal & we provide a roadmap of how to get there
馃У1/View quoted post on X
arxiv.org Linearity of Relation Decoding in Transformer Language Models Much of the knowledge encoded in transformer language models (LMs) may be expressed in terms of relations: relations between words and their synonyms, entities and their attributes, etc. We show... New Paper 馃搫: LMs just want to explain themselves!
When we SFT an LM on explanations of its own behaviors, do they learn to actually introspect, or do they merely imitate the original training distribution?
We find evidence for the former.
Despite training on a static set of explanations from a base model, the SFT-ed model explains its own current behaviors better than the base model鈥檚 behaviors, tracking behavioral drift even when we don鈥檛 explicitly train it to.
We call this introspective coupling: self-explanations track a model鈥檚 own behavior as that behavior changes, and it shows promise in making introspection training a part of scalable post-training pipelines. 馃УView quoted post on X
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs.View quoted post on X
Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [arxiv.org/abs/2606.19317]
