Pinned post
Recent Transformer LMs are very good at using their context for predicting new tokens.
How does this capability arise during training?
We study this in our paper "Birth of a Transformer: A Memory Viewpoint" 馃悾
arxiv.org/abs/2306.00802
How does this capability arise during training?
We study this in our paper "Birth of a Transformer: A Memory Viewpoint" 馃悾
arxiv.org/abs/2306.00802