Twitxy Saved

Nicolas Zucchet

Verified account This account is verified. Learn more

@NicolasZucchet

Postdoc @Stanford, prev. PhD @CSatETH | student researcher @GoogleDeepMind | @Polytechnique
Stanford, CA nicolaszucchet.github.io Joined December 2017
485 Following 1.1K Followers 283 Posts

Latest posts 5

Nicolas Zucchet Verified account @NicolasZucchet
Want to understand LLMs? Start with how neural networks store memories.

This idea deeply shapes how I think about them, so I'm glad to see @heyyalexwang explaining it this clearly, and to see it getting the attention it deserves.

Using this occasion to share some of the recent research I’ve been part of on using this perspective to better understand LLMs:

1. LLMs are extremely good at storing knowledge, but how do they learn to do so? We study this question in arxiv.org/abs/2503.21676, showing that attention circuits need to be created before models can store memories. These attention circuits emerge during training; we study how this depends on data in arxiv.org/abs/2505.17863.

2. The memory setup in the blog, and the standard one more generally, is to have only one possible value, e.g. a next token. In practice, LLMs have to model different possible future tokens; we show in arxiv.org/abs/2608.15448 that such “ambiguity” requires larger networks to store!

3. What happens when a network with a lot of knowledge learns new memories: it catastrophically forgets. This is a nice toy setup to systematically study it and there are some surprising other phenomena that one can study. In arxiv.org/abs/2610.08718 we study spurious forgetting: in some cases there is forgetting in the sense that memories cannot be recalled, but there are still present if probing the network in the right way.

4. One can see post-training as a slight modifications of the memory of the network. We use this perspective in arxiv.org/abs/2610.03529 to give some intuition about how different distillation strategies affect model behavior; this is imo a cool setup to understand some distillation behaviors.

Bonus. From the arguments in the quoted blog, one can heuristically derive scaling laws. I gave a tutorial on that (which reviews some of the work linked in the blog) nicolaszucchet.github.io/Tutorial-scali…; there is even a colab where you can train your own memories and derive their scaling laws!

There are still many open questions related to memories, knowledge and LLMs, and I hope to see a lot more of cool research to do in this direction!
arxiv.org How do language models learn facts? Dynamics, curricula and hallucinations Large language models accumulate vast knowledge during pre-training, yet the dynamics governing this acquisition remain poorly understood. This work investigates the learning dynamics of language...
Alex Wang Verified account @heyyalexwang Oct 16
how much can models remember?

there’s a neat perspective on neural networks that deserves to be more widely known

here’s how you could have invented every layer of the transformer from memory

(1/n) from my new blog post!
View quoted post on X
2 replies 17 reposts 139 likes 8.9K views View on X
Nicolas Zucchet Verified account @NicolasZucchet
LLMs and neural networks catastrophically forget, yet knowledge is often still accessible by probing the hidden states at the right layer or by changing the prompts, even if LLMs cannot directly recall it.

In our new work led by @vedantpalit1008 , we ask why this happens and study the mechanisms responsible for hiding this knowledge.
We consider a super simple setting in which we are sure knowledge is hidden and not lost: recall initially drops during finetuning, then surprisingly increases without any replay. Shared structure is what is hiding behind hiding knowledge!

Happy to have contributed to this work; our results nicely crystallize some intuition that has been around for decades (from the early connectionist days!).
Looking forward to seeing how these insights can help us understand the learning mechanics behind such phenomena in actual LLMs!
Vedant Palit Verified account @vedantpalit1008 Oct 13
Fine-tuning makes language models forget. But sometimes the knowledge isn't gone, just hidden, and it can come back without replay. This is known as spurious forgetting, and it's still unclear what hides it.

We wanted to understand why.
View quoted post on X
3 replies 6 reposts 45 likes 1.8K views View on X
Nicolas Zucchet Verified account @NicolasZucchet
Joining the NeurIPS celebration: our curse of ambiguity paper will be a spotlight 🎉

We show that high entropy tokens need
1. More capacity to store
2. Larger embeddings to be represented
3. More training steps
4. More samples to reduce sampling noise

Scaling laws study performance averaged over all tokens in the data; our work zooms in and sheds some light onto which tokens are harder to learn 🔬
Nicolas Zucchet Verified account @NicolasZucchet Aug 18
We often ask how hard a task is for an LLM to learn, but what about individual tokens?
Our new paper w/ @scott_linderman @hyundongleee digs into that question and reveals a curse of ambiguity: contexts with multiple possible next tokens are harder for LLMs to learn.
View quoted post on X
2 replies 13 reposts 104 likes 6.4K views View on X
Nicolas Zucchet Verified account @NicolasZucchet
Scaling laws have become a central part of how we think about scaling LLMs, but it’s not always easy to build intuition for why they work.

I made a tutorial on:
- scaling laws in practice: classic papers and recent developments
- an intuitive associative memory toy model, where scaling can be predicted from simple intuitions
- a hands-on environment for learning how to use scaling laws

I gave this tutorial at MLSS Tübingen last week, and thought the material was in a good enough shape to share!
x
nicolaszucchet.github.io/Tutorial-scali…