My first PhD paper, 'Sparse Layers are Critical to Scaling Looped Language Models,' is accepted at NeurIPS 2026! Looping & early exits enables adaptive depth; MoE makes it scale. First step toward compute-adaptive LMs.
w/ @jbiloki @edwardjhu @jonathanmay
arxiv.org/abs/2605.09165
w/ @jbiloki @edwardjhu @jonathanmay
arxiv.org/abs/2605.09165
arxiv.org Sparse Layers are Critical to Scaling Looped Language Models Looped language models repeat a set of transformer layers through depth, reducing memory costs and providing natural early-exit points at loop boundaries. However, looped models do not scale as...