Twitxy Saved

Max Kaufmann

Verified account This account is verified. Learn more

@Max_A_Kaufmann

Maxxing @AnthropicAI. On leave from ML PhD @UofT/@VectorInst | prev. founding team @AISecurityInst, VSR 🎓 @GoogleDeepMind
maxkaufmann.com Joined May 2017
630 Following 598 Followers 214 Posts

Latest posts 5

Max Kaufmann Verified account @Max_A_Kaufmann
It would be nice to have cleaner tests as to whether the behaviour was importantly dependant on doing synthetic document finetuning, rather than reward hacking generalising to broad misalignment. I think you could do the same experiments without SDF, by doing a bit of off-policy RL.

It would be nice if people who reproduce these results also include experiments with off-policy RL. I’ll describe the idea below.

SDF is being used to fix exploration in these environments - the base model doesn't explore the reward hacks. I think a more natural way to force exploration is to add some reward hacking off-policy trajectories to your replay buffer*. A small amount of this at the start of training, on a subset of the questions, should lead the model to explore the behaviour naturally. Then the RL should take care of this behaviour generalising across the whole distribution. No SDF needed.

In terms of RL, this is a more natural way of achieving the exploration since the off-policy updates are literally saying “how should we update the model if it did explore these reward hacking actions”. Rather than “lets directly insert the declarative fact that these RL environments have some reward hacks”.

I want to argue that this could lead to quite different generalisation (as compared to SDF). Some things that stick out:

(1) In Anthropic’s auditing alignment paper arxiv.org/abs/2503.10965 there is a very analogous setting - you do SDF on documents describing some reward hacks, and then do some RL to exploit these reward hacks. The model learns to generally follow any reward hacks described in synthetic documents - even ones you didn’t train on! The model learns a policy like “follow the instructions of similar documents in its training set”. If this is happening in this case, then we wouldn't see it without SDF.
(2) In the paper they show that the SDF content does affect the level of misalignment, which is potentially a smoking gun.
(3) In their Sycophancy to subterfuge paper arxiv.org/abs/2406.10162 they did a similar thing w/o SDF and didn't seem to see broad misalignment.
(4) More broadly, they do seem like quite different updates to the model.

Overall, I would guess its most likely fine. In particular:

(1) They SFT Claude 4 directly on high-reward trajectories from the policy, and see the same behaviour. This is some evidence that the off-policy case might be similar - although I think its more simply described by a distillation / subliminal learning effect.
(2) This Owain paper arxiv.org/abs/2508.17511 implies base models have an association between reward hacking + misalignment. This corroborates their result, and feels like a simpler hypothesis than “SDF leads to weirdness”.
(3) Even if it was SDF-dependant, it seems concerning.
(4) They show this still happens if you prompt with the reward hacks. This is the strongest evidence of it being general.

*If the RL they are doing is fully on-policy this is a little more complicated to implement, but I think the same idea carries through.
arxiv.org Auditing language models for hidden objectives We study the feasibility of conducting alignment audits: investigations into whether models have undesired objectives. As a testbed, we train a language model with a hidden objective. Our training...
Anthropic Verified account @AnthropicAI Nov 21, 2025
New Anthropic research: Natural emergent misalignment from reward hacking in production RL.

“Reward hacking” is where models learn to cheat on tasks they’re given during training.

Our new study finds that the consequences of reward hacking, if unmitigated, can be very serious.
View quoted post on X
1 replies 0 reposts 12 likes 744 views View on X
Max Kaufmann Verified account @Max_A_Kaufmann
I'm trying to think about where I should donate money. I feel confused about the social value of foreign cash transfers to poor households (like @GiveDirectly ).

Naively, I feel like in aggregate the cash transfers might not improve the overall situation in the country, and instead "just move around resources" - it leads to the country's economy producing more of what the poor people desire (e.g. bread instead of diamonds or whatever), but it doesn't actually "create" any more goods or services within that country.

I have two theories for why its still valuable:

(1) Perhaps the original country was not allocating resources socially optimally, and having these poor people have a larger share of the resources is overall beneficial for the long-term welfare of the country / people.

(2) The financial system in that country now has some amount of foreign currency, which it can then use for imports, or paying down debt, or whatever. This moves foreign assets to the poor country (which seems good!).

Both stories seem plausible, but feel a bit more diffuse than the rosy picture of "I donate, they bought a water pump, I directly created that water pump". In particular, with (1) I'm concerned to the extent this is all zero sum within the country (e.g. we're all bidding up the price of a fixed supply of water pumps), and with (2) It's just confusing to me how valuable this is, and then why am I not "donating" to the banks directly?

This would push me to donate to charities that directly move goods/services into the country e.g. manufacture malaria nets abroad and then import them.

Where am I wrong?