Can we bootstrap intelligence? Probably! arxiv.org/abs/2609.30063
Thanks @AdityaCowsik @KfirDolev @michaelyli_ @ANourya @noahdgoodman @YoavLevine for the great work!
Thanks @AdityaCowsik @KfirDolev @michaelyli_ @ANourya @noahdgoodman @YoavLevine for the great work!
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?View quoted post on X
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.