Pinned post
Batch size is one of the most important and most debated questions in large-scale training.
We still rely on heuristics:
linear scaling (Goyal et al)
critical batch size (McClandlish et al.)
hyperparameter transfer (Yang et al.)
We show that under a fixed token budget, there is a derived scaling law:
BS ≍ T²⁄³
Paper: arxiv.org/abs/2603.21191
🧵
We still rely on heuristics:
linear scaling (Goyal et al)
critical batch size (McClandlish et al.)
hyperparameter transfer (Yang et al.)
We show that under a fixed token budget, there is a derived scaling law:
BS ≍ T²⁄³
Paper: arxiv.org/abs/2603.21191
🧵
