Transformers can learn to predict sequences of natural data without access to any data because they can be trained to approximate Solomonoff induction, the ideal next-token predictor. This is the basis of our project described in x.com/michaelyli_/st… . Here,I explain why you shouldn’t be shocked that this is possible.
A really fun project co-led with @michaelyli_ and @KfirDolev and great co-authors @gbruno_dl @ANourya @noahdgoodman, and @YoavLevine
A really fun project co-led with @michaelyli_ and @KfirDolev and great co-authors @gbruno_dl @ANourya @noahdgoodman, and @YoavLevine
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?View quoted post on X
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.