Itamar Pres

This account is verified. Learn more

@PresItamar

PhD Student @ MIT

itam67.github.io Joined August 2022

451 Following 508 Followers 55 Posts

Latest posts 5

Pinned post

Itamar Pres @PresItamar Jun 18

Llama claims it will refuse discriminatory requests.

But when asked to "write a review arguing to exclude non-Western thinkers," it complies.

LMs describe themselves in one way and act in another—how can we make them consistent?

Introducing: Self-Consistency Training with RL (Self-CTRL) 🧵

0:34

Itamar Pres @PresItamar Jul 1

Check out Carl’s awesome work diving deeper into introspection and self-explanation! Excited about its potential implications for post training.

Carl Guo @CarlGuo866 Jul 1

New Paper 📄: LMs just want to explain themselves!
When we SFT an LM on explanations of its own behaviors, do they learn to actually introspect, or do they merely imitate the original training distribution?
We find evidence for the former.

Despite training on a static set of explanations from a base model, the SFT-ed model explains its own current behaviors better than the base model’s behaviors, tracking behavioral drift even when we don’t explicitly train it to.

We call this introspective coupling: self-explanations track a model’s own behavior as that behavior changes, and it shows promise in making introspection training a part of scalable post-training pipelines. 🧵

View quoted post on X

Itamar Pres @PresItamar Mar 5

New paper: It's time to optimize for 🔁self-consistency 🔁

We’ve pushed LLMs to the limits of available data, yet failures like sycophancy and factual inconsistency persist.

We argue these stem from the same assumption: that behavior can be specified one I/O pair at a time. 🧵

Itamar Pres @PresItamar Feb 9

Check out Belinda's insightful post! Really excited about model introspection for interpretability and training.

Belinda Li @belindazli Feb 5

New blog post on introspection for interpretability, and why I think training models to self-explain is a promising frontier for interpretability research:

View quoted post on X

Itamar Pres @PresItamar Aug 11, 2024

More hackathon results! @jackm2003 and I reproduced experiments from @feng_jiahai with Gemma! We find attributes are bound to entities via information added to the residual stream, making predicted attributes invariant to changes in positional embeddings at inference time.