Pinned post
As we move from solving competition maths to Erdos problems to discovering new mathematical structures, we scale inference compute accordingly; models think longer, sample alternative pathways, and synthesize those attempts into a refined output. Yet during training, they are trained to think with only sequential compute, where each generation gets rewarded in isolation.
Excited to share our work SPIRAL: an RL framework that trains models to use all primitives of inference compute — sequential, parallel, and aggregative — via end-to-end learning using solely the reward of the final output. We show that this naturally required standard RL to train a model how to aggregate and set RL to train a model to sample parallel traces optimal for downstream aggregation. As a result, the model learns the optimal way to allocate its test-time compute. Our early results show that SPIRAL achieves significantly better performance as all three primitives of inference compute are scaled. Work with @jubayer_hamid (co-lead), @michaelyli_ , @oshaikh13, @yoonholeee, @DorsaSadigh, @chelseabfinn, @noahdgoodman.
Check out our work: arxiv.org/abs/2606.23595
Excited to share our work SPIRAL: an RL framework that trains models to use all primitives of inference compute — sequential, parallel, and aggregative — via end-to-end learning using solely the reward of the final output. We show that this naturally required standard RL to train a model how to aggregate and set RL to train a model to sample parallel traces optimal for downstream aggregation. As a result, the model learns the optimal way to allocate its test-time compute. Our early results show that SPIRAL achieves significantly better performance as all three primitives of inference compute are scaled. Work with @jubayer_hamid (co-lead), @michaelyli_ , @oshaikh13, @yoonholeee, @DorsaSadigh, @chelseabfinn, @noahdgoodman.
Check out our work: arxiv.org/abs/2606.23595
The most capable reasoning systems in AI scale inference compute along several axes: sequential compute to think longer, parallel compute to sample many independent attempts, and aggregative compute to synthesize prior traces into a new improved one. But during training, we only optimize how models use sequential compute. This creates a fundamental mismatch between how we ultimately deploy these systems and how we train them, leaving much of search and synthesis unoptimized.View quoted post on X
We introduce SPIRAL, an RL framework for making all inference-compute primitives end-to-end learnable: models learn to coordinate sequential, parallel, and aggregative reasoning using only the reward of the final output. Work with @ifdita_hasan (co-lead), @michaelyli_ , @oshaikh13 , @yoonholeee , @DorsaSadigh , @chelseabfinn , @noahdgoodman 🧵