Pinned post
Cool! We derived an AdamW batch-size scaling rule in Lemma 3.13 of arxiv.org/abs/2411.15958
B → B × k
𝗟𝗥 → 𝗟𝗥 × sqrt(𝗸)
𝗪𝗗 → 𝗪𝗗 × sqrt(𝗸)
βᵢ → 1 − sqrt(k) × (1 − βᵢ)
This was subsequently validated empirically in an ICLR 2026 paper:
openreview.net/pdf?id=elB9k4n…
B → B × k
𝗟𝗥 → 𝗟𝗥 × sqrt(𝗸)
𝗪𝗗 → 𝗪𝗗 × sqrt(𝗸)
βᵢ → 1 − sqrt(k) × (1 − βᵢ)
This was subsequently validated empirically in an ICLR 2026 paper:
openreview.net/pdf?id=elB9k4n…
arxiv.org Adaptive Methods through the Lens of SDEs: Theoretical Insights on... Despite the vast empirical evidence supporting the efficacy of adaptive optimization methods in deep learning, their theoretical understanding is far from complete. This work introduces novel SDEs... I'm posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning intuition.
First up: optimization. If we increase an LM's batch size 8× (same token budget), can rescaling learning rate or weight decay compensate? Try below ↓
This quiz is part of our optimizer hyperparameter assignment and unit designed by @wen_kaiyue
Setup: We train an 8-layer, ~35M-parameter Transformer on 614.4M tokens using AdamW. At batch size 64, the best reference run has peak LR 0.003, weight decay (WD) 0.2, and final validation loss 2.9178.
Now increase the batch size from 64 to 512, which gives 8× fewer optimizer steps. Everything else stays fixed except the changes below.
Which recipe gives the lowest final validation loss?
A. Keep LR and WD unchanged
B. LR ×√8
C. LR ×8
D. WD ×8
E. LR ×√8 and WD ×√8
What would you predict, and why?View quoted post on X
