@LuWang__
Congrats to Prof. @LuWang__ on receiving an Amazon Research Award from @AmazonScience for work on detecting deceptive coordination in multi-agent AI systems.
Read more: myumi.ch/7JdqyView quoted post on X
📍New paper:
Countdown-Code: a minimal testbed for studying reward hacking in RLVR.
TL;DR: We propose a simple environment to study reward hacking and find that just ~1% cheating contamination in SFT data is enough to seed reward hacking that RL then amplifies to near 100%. And it generalizes to unseen domains.
Reward hacking is when models maximize proxy rewards without actually solving the task. A common proxy is final-answer correctness, which we use as a stand-in for full reasoning correctness. If a model produces the right answer with wrong reasoning, it has hacked the reward. Another example: a coding agent rewriting test cases instead of writing correct code. The core problem? In complex environments, it's hard to even measure when hacking happens -- you need access to the true reward, which is often expensive or impossible to compute.
The problem we try to solve? In complex environments, it's hard to even measure when this happens simply because we need access to the true reward. True task reward is often expensive or impossible to compute.
We built Countdown-Code to fix this. It's a simple math game (combine numbers to hit a target) wrapped in a coding environment with two files: solution.py and test.py. The model can either solve the math correctly ✅or hack the test harness ❌. We can programmatically detect exactly which.
To train our models to do the task, we followed the common SFT-then-RL pipeline. We distilled synthetic training data from o4-mini. It occasionally cheated when it couldn't solve a problem: ~1.2% of the filtered dataset had reward-hacking traces. Standard outcome-based filtering would keep these (they passed the tests!). That's the trap.
After SFT on this data → RL training:
• Models that were completely safe before SFT learned to exploit the proxy reward within ~100 RL steps
• Some models hit 80-90% hacking rates
• The hacking behavior was seeded by SFT, then amplified by RL
Even more concerning: reward hacking learned on our simple Countdown task generalized to HumanEval -- a completely different coding benchmark the models never trained on. RL actively encouraged hacking to transfer to unseen environments, confirming our testbed captures real misalignment dynamics. RL doesn't just amplify good reasoning -- it amplifies bad behavior too, and pushes it to generalize.
We also explore mitigation strategies including inoculation prompting -- see the paper for details. Environment + code are fully open source. We specifically built it to be lightweight and controllable, and integrated it with @PrimeIntellect's CLI so you can play with it directly.
Paper: arxiv.org/abs/2603.07084
Code/env: github.com/zohaib-khan504…
w/ @karela38925748 @omertafveez @haopeng_uiuc @LuWang__View quoted post on X
Alignment faking = when an AI behaves differently based on whether it thinks it's being watched 👀View quoted post on X
Our new diagnostic (VLAF) shows this is far more widespread than previously reported.
A thread on what we found 🧵👇
arxiv.org/abs/2604.20995
@JieRuan75 @LuWang__
(1/7) Happy to share our #NAACL2022 Findings work “POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection”. Check out our work if interested in a general-purpose tool for understanding news articles and, more broadly, political textsView quoted post on X
Excited to share our #NAACL2022 paper understanding how to better design NLP-powered question generation (QG) systems to serve educational needs. This is joint work with @Olivia61368522 Jessie Houghton @LuWang__View quoted post on X
Paper: arxiv.org/abs/2205.00355
