I couldn’t be prouder to share this. 🎉
Our work on Quantile Reward Policy Optimization (QRPO) for LLM RL‑finetuning bridged deep theory and large‑scale practice:
* Theory first. We cracked the partition‑function “intractable” myth, reframing it with moment‑generating functions and a family of tractable reward transformations.
* Scale matters. Thousands of controlled chat & code runs put QRPO up against DPO, REBEL, and SimPO, testing length bias, sample‑probability shifts, β‑sensitivity, and more.
I hope QRPO sparks new directions for aligning LLMs straight to reward signals - while staying simple, stable, and effective in both offline/off‑policy and online RL.
There’s still plenty to explore (appendix lovers, rejoice!), but the results speak for themselves.
Huge thanks to my brilliant collaborators @SkanderMoalla and @caglarml - couldn’t have done it without you. 🙏
📰 Paper: arxiv.org/abs/2507.08068
🧑💻 Code: github.com/CLAIRE-Labo/qu…
🌐 Blog: claire-labo.github.io/quantile-rewar…
Star the repo, share feedback, and let’s keep pushing LLM alignment forward! 🚀
Our work on Quantile Reward Policy Optimization (QRPO) for LLM RL‑finetuning bridged deep theory and large‑scale practice:
* Theory first. We cracked the partition‑function “intractable” myth, reframing it with moment‑generating functions and a family of tractable reward transformations.
* Scale matters. Thousands of controlled chat & code runs put QRPO up against DPO, REBEL, and SimPO, testing length bias, sample‑probability shifts, β‑sensitivity, and more.
I hope QRPO sparks new directions for aligning LLMs straight to reward signals - while staying simple, stable, and effective in both offline/off‑policy and online RL.
There’s still plenty to explore (appendix lovers, rejoice!), but the results speak for themselves.
Huge thanks to my brilliant collaborators @SkanderMoalla and @caglarml - couldn’t have done it without you. 🙏
📰 Paper: arxiv.org/abs/2507.08068
🧑💻 Code: github.com/CLAIRE-Labo/qu…
🌐 Blog: claire-labo.github.io/quantile-rewar…
Star the repo, share feedback, and let’s keep pushing LLM alignment forward! 🚀
arxiv.org Quantile Reward Policy Optimization: Alignment with Pointwise... Aligning large language models with pointwise absolute rewards has so far required online, on-policy algorithms such as PPO and GRPO. In contrast, simpler methods that can leverage offline or... 🚀 Big time! We can finally do LLM RL fine-tuning with rewards and leverage offline/off-policy data!
❌ You want rewards, but GRPO only works online?
❌ You want offline, but DPO is limited to preferences?
✅ QRPO can do both!
🧵Here's how we do it:View quoted post on X
