Cameron R. Wolfe, Ph.D.

I just published my complete guide to reinforcement learning for LLMs. It's a single, standalone resource for understanding RL from first principles to frontier research.

Read it here: cameronrwolfe.substack.com/p/llm-rl

The topics covered are as follows…

RL Fundamentals:
- A general framework for RL (policy, states, actions, transition functions, environment, rewards, trajectories, etc.)
- Core concepts of RL (returns, discounting, action / state / trajectory probabilities, value / advantage functions).
- Formulating and estimating the learning objective for RL.

Basic of RL for LLMs:
- Token-level (MDP) vs. completion-level (bandit) formulations.
- Outcome versus process rewards.
- Value estimation via value models / critics.
- KL divergence, RL setups (RLHF, RLVR, infra, etc.), importance sampling.

Basic Policy Gradients:
- Deriving the RL objective / vanilla policy gradient (VPG) from first principles.
- Different policy-gradient formulations (trajectory returns, reward-to-go, baselines, value functions, and advantages).
- Why policy gradient estimates have high variance (and how baselines help).
- Implementation of VPG.

REINFORCE and its variants:
- REINFORCE as a Monte Carlo implementation of the VPG.
- Completion-level vs. token-level implementations of REINFORCE.
- Reward baselines and KL-regularized rewards.
- RLOO, REINFORCE++ and other critic-free policy gradient algorithms.

Actor-Critic Methods:
- Training value models / critics alongside the policy to estimate advantages.
- Trust Region Policy Optimization (TRPO) and KL-constrained policy updates.
- How PPO simplifies TRPO with a clipped surrogate objective.
- Full PPO details (clipping, advantage estimations, value loss, KL, policy ratios, multi-epoch optimization, etc.) + implementation.
- Generalized Advantage Estimation (GAE) + implementation.
- The memory and computational overhead of actor-critic algorithms.

GRPO:
- Group Relative Policy Optimization (GRPO) as a simpler alternative to PPO.
- Group-relative advantage estimates + implementation.
- Why removing the critic is helpful.
- Limitations of vanilla GRPO algorithm.
- Dr. GRPO, DAPO, GSPO, and other recent improvements.

Recent research topics:
- Online versus offline RL.
- Rubric-based RL.
- RL scaling laws.
- Agentic RL + world modeling.

This post is a synthesis of my writing / learning on RL for over a year. I hope it’s helpful!

I just published my deep dive on score centering, a simple technique for stabilizing LLM RL that is (allegedly) popular in frontier labs…

TL;DR. LLM RL has gotten harder to keep stable over time as we work with large MoEs, disaggregated / async architecture, agentic tasks, etc. Score centering provides a clean explanation for why RL becomes unstable under training-inference mismatch (TIM) and proposes a simple, additive correction to the policy gradient that solves this problem. Full post is here: cameronrwolfe.substack.com/p/score-center…

(1) Training-inference mismatch (TIM). Modern RL infra often separates rollout generation and policy updates into two engines—the sampler and trainer. These engines should represent the same policy, but practical implementation details cause their token probabilities to differ.

Such mismatches can arise from numerical precision, quantization, different GPU kernels, or asynchronous training with stale rollouts. These differences may seem small, but they can introduce systematic biases into policy-gradient updates that accumulate and cause instability.

(2) Gradient drift. To understand why TIM causes instability, we can consider a simple setup where every completion receives a constant reward. In theory, the expected policy gradient should be zero because there is no information about which completions are better than others.

Under TIM, however, we can show that this expected gradient is not zero—we have a spurious learning signal or bias in our policy gradient update. This spurious learning signal is referred to as gradient drift. It pushes the trainer toward the sampler regardless of the reward signal.

(3) Feedback loop. The downside of drift does not stop here. In online RL, we constantly refresh our sampler from the trainer, creating a feedback loop. We apply a biased update to our trainer, update the weights of the sampler after this biased update, then use this sampler to sample new rollouts. Errors can accumulate as we continually bias our trainer and sampler.

(4) Score centering. Drift exists because the expected token-level score under the sampler is not necessarily zero when the sampler and trainer differ. Score centering fixes this by subtracting the expected score under the sampler from the score of each sampled token. This additive correction ensures that the centered score has zero expectation at every prefix.

(5) Efficient implementation. Computing the expected score seems expensive because it requires considering every possible next token in the vocabulary at every prefix.The paper shows that we can approximate this correction using only the sampler's top-k (where k = 32 or 128) token probabilities. The remaining probability mass is approximated using the trainer's distribution, and the correction can be reformulated to depend only on the top-k tokens.

(6) Importance sampling. Notably, TIM is usually addressed with importance sampling methods, but these are not mutually exclusive with score centering! The authors of the paper actually show that the best performance is often achieved with a combination of score centering with truncated or masked importance sampling. The addition of importance sampling tends to help deal with cases of extreme mismatch; e.g., async infra with very stale rollouts.

Coming out tomorrow morning. This blog was quite a challenge (but also fun!). My recent blogs have focused on breadth, so I wanted to go deeply into a topic this time. Score centering is interesting and a great topic to focus on given its (alleged) popularity in the labs.

The score centering paper is great. Partly because it works and partly because the math is clean / fun. It's simple enough to follow but nuanced enough to make it rewarding. Working through the math yourself and getting everything to click really is satisfying and nostalgic.

My talk from AI Engineer NYC is now on Youtube!

The talk covers a simple roadmap for agent evals. It focuses on tasks that are subjective / non-verifiable, but the core concepts can be applied to any eval. Key topics that it touches upon are:

- Human evals
- Quality guidelines
- LLM judges / rubrics
- Calibration (both humans and judges)
- Data sampling techniques
- Statistical tools for better evaluation
- A/B testing and production monitoring
- Continual evolution of benchmarks

Here is the link to the talk: youtube.com/watch?v=zLc8ze…

The slides are here: docs.google.com/presentation/d…

To go along with the talk, I also create a full written guide that covers all of the topics from the talk in more detail and links out to further reading resources: cameronrwolfe.substack.com/p/subjective-e…