Twitxy Saved

Anthony Zhan

Verified account This account is verified. Learn more

@probablynotaz9

Stanford, CA Joined April 2024
166 Following 144 Followers 16 Posts

Latest posts 4

Anthony Zhan Verified account @probablynotaz9
Code for QuasiMoTTo is out! github.com/probablyabot/q…
github.com/probablyabot/v…

We're releasing both the RL training code, which builds on TRL, and the pass@k eval, which builds on our vllm fork.

In both settings, QuasiMoTTO is a simple drop in replacement for iid sampling. Correlated samples -> better coverage -> more efficient pass@k and RL training!
github.com GitHub - probablyabot/quasimotto Contribute to probablyabot/quasimotto development by creating an account on GitHub.
Michael Y. Li Verified account @michaelyli_ Jul 2
You're wasting FLOPs when scaling inference compute: by independently sampling parallel attempts, you burn compute rediscovering the same solutions.

Introducing QuasiMoTTo: we scale parallel sampling with correlated samples instead! These samples have higher coverage, are marginally exact draws from the LLM, and can be generated in parallel.

Result: same performance with 25-47% fewer samples in test-time scaling + 50% fewer training steps in RL!

In our new paper, we explore the design space of correlated samplers. Work with co-authors @probablynotaz9 (co-lead), @gandhikanishk, @noahdgoodman, and Emily Fox!
View quoted post on X
1 replies 1 reposts 10 likes 852 views View on X
Anthony Zhan Verified account @probablynotaz9
A lot of thought goes into designing harnesses, algorithms, objectives, etc. for parallel test-time compute and GRPO-style policy gradients for LLMs. Sampling is usually an afterthought: just use a bunch of independent rollouts.

However, this misses out on a rich design space of non-iid samplers, particularly from Quasi-Monte Carlo methods. You can think of these as choosing a joint distribution on [0,1]^k such that each sample is marginally uniform, but the k samples collectively cover [0,1] better than iid.

In QuasiMoTTo, we revisit these classic ideas, showing how they fit into the age of LLM test-time compute (pass@k, online RL). On tasks like Maze/Sudoku, swapping out iid sampling with QMC accelerates RL training (by lowering the fraction of zero-variance groups) and even saturates an upper bound on pass@k.

The idea of coordinating a group of rollouts by simply sharing the sampling noise is really elegant IMO, and there's lots of potential directions to take this idea. Super fun working on this project with @michaelyli_ !

Also stay tuned for the codebase dropping in a few days🫡
Michael Y. Li Verified account @michaelyli_ Jul 2
You're wasting FLOPs when scaling inference compute: by independently sampling parallel attempts, you burn compute rediscovering the same solutions.

Introducing QuasiMoTTo: we scale parallel sampling with correlated samples instead! These samples have higher coverage, are marginally exact draws from the LLM, and can be generated in parallel.

Result: same performance with 25-47% fewer samples in test-time scaling + 50% fewer training steps in RL!

In our new paper, we explore the design space of correlated samplers. Work with co-authors @probablynotaz9 (co-lead), @gandhikanishk, @noahdgoodman, and Emily Fox!
View quoted post on X
9 replies 6 reposts 36 likes 11K views View on X
Anthony Zhan Verified account @probablynotaz9
Excited to present this at ICML next week! ✈️🇰🇷

I'll be there in person from Monday to Friday. If you're also in Seoul and interested in RL/diffusion or more generally LLM post-training/inference, let's chat :)
Anthony Zhan Verified account @probablynotaz9 May 9
🚨 Solo-author ICML paper alert 🤫

Ever wanted to post-train your diffusion LLM with good old policy gradients, without having to deal with ELBOs or surrogates?

In Simple Policy Gradients for Reasoning with Diffusion Language Models, we show how to make this tractable in a straightforward way.

Our framework, Amortized GRPO (AGRPO), lets the model learn from unbiased PG updates via timestep estimation, naturally aligning with dLLM inference while remaining efficient + scalable.

Paper: arxiv.org/abs/2510.04019
Code: github.com/probablyabot/a…

1/n
View quoted post on X
0 replies 1 reposts 19 likes 2.3K views View on X
Anthony Zhan Verified account @probablynotaz9
🚨 Solo-author ICML paper alert 🤫

Ever wanted to post-train your diffusion LLM with good old policy gradients, without having to deal with ELBOs or surrogates?

In Simple Policy Gradients for Reasoning with Diffusion Language Models, we show how to make this tractable in a straightforward way.

Our framework, Amortized GRPO (AGRPO), lets the model learn from unbiased PG updates via timestep estimation, naturally aligning with dLLM inference while remaining efficient + scalable.

Paper: arxiv.org/abs/2510.04019
Code: github.com/probablyabot/a…

1/n