Pinned post
We found a problem with on-policy self-distillation. The student makes mistakes early in a rollout, and the teacher has to continue from that flawed prefix, so its learning signal gets a lot weaker.
So we asked, "What if the teacher could step in before those mistakes happen?" Our method, Speculative Self-Distillation, lets the student generate while the teacher intervenes when their predictions diverge a lot.
We got better accuracy and training efficiency with our results, plus less forgetting in our tests. See @talaei_shayan’s thread for more!
Thanks to my amazing co-authors @chewing_a_gum, @jonashubotter, Amin Saberi and @Azaliamirh.
So we asked, "What if the teacher could step in before those mistakes happen?" Our method, Speculative Self-Distillation, lets the student generate while the teacher intervenes when their predictions diverge a lot.
We got better accuracy and training efficiency with our results, plus less forgetting in our tests. See @talaei_shayan’s thread for more!
Thanks to my amazing co-authors @chewing_a_gum, @jonashubotter, Amin Saberi and @Azaliamirh.
1:10
Introducing Speculative Self-Distillation (SSD, NeurIPS 2026), which pushes the accuracy–efficiency–forgetting frontier of self-distillation.View quoted post on X
In SSD, student and teacher (the same model with additional information) write the rollouts together. Like a private tutor, the teacher lets the student write and takes over only when the student drifts too far. As a result, SSD trains on rollouts that stay as close to the student's own writing (on-policy) as possible while the teacher keeps them on track.
📉 38–69% fewer training tokens than on-policy, matching its accuracy
⏱️ Reaches on-policy accuracy in ~130 vs ~414 min (Science Q&A)
🧠 Forgets less in a stress test: −1.4 vs −4.5 points for on-policy