This is insane
4-bit all the way from pre-training to post-training.View quoted post on X
Pre-training done in just 68 hours.
Our model factory keeps shipping. Proud of our team!
Latest posts 5
The scarcity of compute led to all labs converging on the same recipe prescribed by compute-optimal scaling laws. As we transition to the data-optimal regime things will start looking very different (and maybe more human-like).
Humans are ~1,000,000x more sample efficient than LLMs, and there are no good explanations.
We’ve found that a big part of the answer is that LLMs are *massively* underparameterized by many orders of magnitude. The brain has 100T synapses learning from ~200M tokens of language. Chinchilla prescribes a 1T-parameter model on 20T tokens. That’s 7 OOMs apart in parameters per token.
After a lot of painstaking work, we present new data-constrained scaling laws and learning algorithms to scale to (extrapolated) quadrillion-parameter transformers.
w/ @akshayvegesna, @EthanBaron78553, @asatheesh6561
Full paper led by @EthanBaron78553, with @ShmuelBerman, @andrewgwilsView quoted post on X
agent swarms are pretty cool
Argon is heavier than carbon, so why did they name the better (and presumably bigger) model carbon? Is it better AND smaller?
Noise tracks are reset to 30M tokens. The forward-only record is now 3.409
Also a new track: any architecture you want, still no backprop. Go wild!
github.com/qlabs-eng/noise
Also a new track: any architecture you want, still no backprop. Go wild!
github.com/qlabs-eng/noise
github.com GitHub - qlabs-eng/noise: Speedrun for pretraining without backpropagation Speedrun for pretraining without backpropagation. Contribute to qlabs-eng/noise development by creating an account on GitHub. I think the last month might have been the biggest month in (zeroth-order) optimization history.
In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art.
Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works!
Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M.
Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out.
Expect a lot more from us over the coming weeks.View quoted post on X