Twitxy Saved

Nourya

Verified account This account is verified. Learn more

@ANourya

Stanford BS/MS | ML Research
Joined January 2023
40 Following 103 Followers 3 Posts

Latest posts 1

Pinned post

Nourya Verified account @ANourya
so excited to see this out in the world!
Michael Y. Li Verified account @michaelyli_ Sep 25
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?

Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.

A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
View quoted post on X