Twitxy Saved

Suhas Kotha

@kothasuhas

cs phd @ stanford
kothasuhas.github.io Joined May 2020
370 Following 1.4K Followers 222 Posts

Latest posts 5

Pinned post

Suhas Kotha @kothasuhas
Since compute grows faster than the web, we think the future of pre-training lies in the algorithms that will best leverage ♾ compute

We find simple recipes that improve the asymptote of compute scaling laws to be 5x data efficient, offering better perf w/ sufficient compute
10 replies 84 reposts 450 likes 157K views View on X
Suhas Kotha @kothasuhas
nice, solid data efficiency win :))
x.com/xingyudang/sta…
Xingyu Dang Verified account @xingyudang Nov 12
Compute grows fast but pretraining corpora don’t, so data-constrained optimization matters!

We propose Training-Time Testing (T³): each step takes a trial step on half the batch, tests it on the other half, and updates with the meta-gradient.

It achieves SOTA on SlowRun!
View quoted post on X
0 replies 2 reposts 24 likes 1.7K views View on X
Suhas Kotha @kothasuhas
interesting how pretraining loss may hide finetuneability
x.com/CatherineeLii/…
Catherine Li @CatherineeLii Nov 3
Muon achieves lower loss than AdamW after pretraining, but does its advantage persist after fine-tuning?

Surprisingly, the answer depends on pretraining data scale: beyond a sufficiently large pretraining data budget, AdamW outperforms Muon after fine-tuning!
View quoted post on X
1 replies 1 reposts 44 likes 2.8K views View on X
Suhas Kotha @kothasuhas
check out darrow’s simple explainer on edge of stability as taught in CS312!
Darrow Verified account @darrowoah Oct 30
This week in CS 312, we explored one of the most unintuitive concepts in pretraining: sharpness

In this post, I'll explain what sharpness is, how sharpness appears in real training, and our unanswered questions about sharpness x.com/i/article/2116…
View quoted post on X
0 replies 5 reposts 50 likes 6.7K views View on X
Suhas Kotha @kothasuhas
pretty cool study by ahmed/nick about the compute/data tradeoffs of rephrasing! im most excited about the demo where rephrasing achieves self-improvement during data-constrained pre-training (with just a few SFT examples for an initial rephraser)
x.com/AhmedSQRD/stat…
Ahmed Ahmed Verified account @AhmedSQRD Oct 30
Synthetic rewrites are now standard to improve data quality and when data is limited. But how much compute should you spend rewriting? And can a model trained on rewrites beat the rewriting model?

We fit scaling laws to answer both:

1/🧵
View quoted post on X