Samip

This account is verified. Learn more

@industriaalist

cofounder Q Labs (qlabs.sh)
solving data efficiency and generalization

sf samipdahal.xyz Joined June 2016

121 Following 8.2K Followers 497 Posts

Latest posts 5

Pinned post

Samip @industriaalist Nov 2

Humans are ~1,000,000x more sample efficient than LLMs, and there are no good explanations.

We’ve found that a big part of the answer is that LLMs are *massively* underparameterized by many orders of magnitude. The brain has 100T synapses learning from ~200M tokens of language. Chinchilla prescribes a 1T-parameter model on 20T tokens. That’s 7 OOMs apart in parameters per token.

After a lot of painstaking work, we present new data-constrained scaling laws and learning algorithms to scale to (extrapolated) quadrillion-parameter transformers.

w/ @akshayvegesna, @EthanBaron78553, @asatheesh6561
Full paper led by @EthanBaron78553, with @ShmuelBerman, @andrewgwils

Samip @industriaalist 5d

We have some *bold* ideas on solving compression and closing the sample efficiency gap between humans and models.

Who can intro me to @ilyasut?

Samip @industriaalist Nov 1

we should expect to see decades of algorithmic progress in deep learning in the next year. almost no one takes this seriously.

exponentials are real and it’s becoming very clear to us internally.

Samip @industriaalist Oct 22

also tagging the GOATs @geoffreyhinton @ylecun to update their prior!

Samip Verified account @industriaalist Oct 22

I think the last month might have been the biggest month in (zeroth-order) optimization history.

In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art.

Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works!

Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M.

Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out.

Expect a lot more from us over the coming weeks.

View quoted post on X

Samip @industriaalist Oct 22

I think the last month might have been the biggest month in (zeroth-order) optimization history.

In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art.

Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works!

Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M.

Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out.

Expect a lot more from us over the coming weeks.