How do you teach an LLM a new document without it forgetting what it already knows? 📚 Check out Speculative Self-Distillation, a #NeurIPS2026 paper from @talaei_shayan, @chewing_a_gum, @ckodser and @Azaliamirh's group. The student writes, and the teacher steps in only where it matters.
Introducing Speculative Self-Distillation (SSD, NeurIPS 2026), which pushes the accuracy–efficiency–forgetting frontier of self-distillation.View quoted post on X
In SSD, student and teacher (the same model with additional information) write the rollouts together. Like a private tutor, the teacher lets the student write and takes over only when the student drifts too far. As a result, SSD trains on rollouts that stay as close to the student's own writing (on-policy) as possible while the teacher keeps them on track.
📉 38–69% fewer training tokens than on-policy, matching its accuracy
⏱️ Reaches on-policy accuracy in ~130 vs ~414 min (Science Q&A)
🧠 Forgets less in a stress test: −1.4 vs −4.5 points for on-policy