Twitxy Saved

Sham Kakade

Verified account This account is verified. Learn more

@ShamKakade6

Harvard Professor.
Full stack ML and AI.
Co-director of the Kempner Institute for the Study of Artificial and Natural Intelligence.
shamulent.github.io Joined December 2018
597 Following 20K Followers 552 Posts

Latest posts 5

Sham Kakade Verified account @ShamKakade6
How far can quadratics take us in LLM pretraining? Surprisingly far.
Excited to give a plenary at #ICML2026 on how a simple, solvable quadratic model predicts real pretraining — critical batch size, LR decay, batch-size scaling, and what momentum does (and doesn't) do.
#KempnerInstitute Co-Director Sham Kakade’s upcoming #ICML2026 plenary (7/8 @ 8:30am KST) will spotlight a broad science-of-pretraining effort—showcasing how a remarkably simple model can accurately forecast pretraining outcomes. Learn more: bit.ly/4eNVmSp @shamkakade6
View quoted post on X
3 replies 9 reposts 82 likes 11K views View on X
Sham Kakade Verified account @ShamKakade6
Momentum accelerates training, but are the savings compute or serial runtime? New work: we prove compute-efficiency (CE: total compute to reach a target loss, not serial steps) lower bounds for stochastic Heavy Ball (HB) and Accelerated SGD (ASGD) [Kidambi et al., 2018]
3 replies 19 reposts 82 likes 66K views View on X
Sham Kakade Verified account @ShamKakade6
1/8 Introducing Recurrent Transformer (RT). At 300M params, RT improves validation CE over standard Transformers. The best RT model is only 6 layers, but wider at 2048 — beating deeper 12- and 24-layer Transformers by trading depth for width.