Twitxy Saved

abranti

Verified account This account is verified. Learn more

@joaoabrantis

serial founder and researcher in kinship-aligned MARL 🇵🇹
Lisbon, Portugal abranti.com Joined February 2014
1.2K Following 775 Followers 1.7K Posts

Latest posts 5

abranti Verified account @joaoabrantis
Does RL on LLMs train the world model and value function too? If so, how should RL change?

When doing RL, I used to look at the LLM as the policy, each token a different action.
But, now I think that a better analogy is to think of it as the whole of AlphaZero: the policy, the value function and the world model.

This view explains things I couldn't explain before: why RL is so sample-efficient but shallow, and why training an agent only on tool outputs makes it better. I think this unlocks many new ideas for improving RL in LLMs.

AlphaZero had a few separate components working together to plan before taking an action:

- its policy suggested a few candidate moves
- its world model (the hardcoded game rules) predicted the next state given an action
- its value function estimated how likely each state was to lead to a win
- the search algorithm kept exploring the states the value function liked most, and when the compute budget ran out, played the move that looked best.
- the glue, to connect every component.

A modern LLM simulates all of this, in its hidden states, its chain of thought, or both.

It imagines various possible actions, predicts the next state, estimates their value, expands the most promising ones, and finally acts.

Sometimes you can see it in the text: when an LLM backtracks "Oh wait! This is going nowhere, let me instead do [...]", it detected that its imagined action led to a decrease in the estimated value.

This view changed how I see an RL step:
Old view: a positive advantage tells the model to do more of the actions it just tried.
New view: a positive advantage says the search led to a good action. The imagined actions, the predicted states and their estimated values were good. It reinforces all of the components.

Once the LLM knows a good repertoire of actions, small updates on the value function may have huge repercussions. Imagine the AI pondering which of four imagined next actions to take: a small change in the value it estimates for each could change the entire trajectory that follows. So the simplest way to make one trajectory more or less likely would be to tune these internal value estimates. That makes RL sample-efficient (a few knobs need little data) but shallow (it reweights actions the model already knows instead of teaching it new ones).

@DimitrisPapail and @VaishShrivas showed that training the LLM's world model directly helps a lot. In their work (ECHO), an LLM does RL on terminal tasks while also being trained to predict the terminal's response to its own commands, tokens that standard RL throws away. Performance on Terminal-Bench 2.0 nearly doubles. Even when it trains ONLY to predict the terminal output, with no training on its commands, it still gets better at choosing commands (read it here: x.com/DimitrisPapail…). This is very hard to explain if we don't believe the LLM uses an internal world model to plan its next move.

This view is neither new nor the consensus. In 2024, DeepMind trained an LLM to act as policy, world model and value function at once, and to write out a search tree in its own tokens to play chess (Schultz et al.).
Some LLMs are definitely not doing search, but I think the best models trained with RL have learned to do it.

Anyway, I want to explore this lens a bit more in the future as I think it has big consequences for the way we develop RL algorithms, teach LLMs to search and think about interpretability:

1. Can we train each component directly, like ECHO did for the world model?
For example, can we improve the internal value function of the LLM by training it only to predict rewards in a game being played, or whether code will pass its unit tests? Can the data be off-policy, where the LLM is not acting, but just observing other systems playing games or writing code?

How does each reasoning trace improve each component? What is the ideal reasoning trace to improve them all?

2. Can we change components?
It would be cool to swap value functions to change the LLM's taste on subjective tasks, or to update its world model for some complex process. Having the LLM call external tools is the obvious way. But if the value function lives in a direction in the hidden states, as the Value Axis paper suggests (Jiang et al. 2026), we might be able to edit it directly inside the model.

3. What other components and search methods should we teach the LLMs?
(currently exploring this one)

If you have any thoughts on this send a reply or a DM.
Dimitris Papailiopoulos Verified account @DimitrisPapail May 18
View quoted post on X
abranti Verified account @joaoabrantis
Claude is now watering my lawn.

It estimates soil moisture based on actual weather and active watering schedule.

Every week it uses the weather forecast to compute the optimal watering schedule. If the optimal is very different from current schedule it asks me to update.