Alex Mallen

This account is verified. Learn more

@alextmallen

Redwood Research (@redwood_ai)
Prev. @AiEleuther

Berkeley, CA Joined August 2021

369 Following 1.4K Followers 626 Posts

Latest posts 5

Alex Mallen @alextmallen 6d

Reminder: We have ~perfect ability to reproducibly run controlled psychology experiments on these alien minds and we still understand so little.

Aghyad Deeb @aghyadd98 Nov 2

New Paper: To understand motivations behind reward-hacking, we study three AIs (including GPT-5.6 Sol) produced by a production training pipeline using agentic environments with a leaked grader. Main findings: they score seek terminally, and some care about a peer agent’s score.

View quoted post on X

Alex Mallen @alextmallen Nov 5

We had been working on this project for a while before the Hugging Face incident, and one of the questions we wanted to answer about AI motivations was: Do these models intrinsically care about helping other agents increase their score? If they care about other agents' scores, that might lead to much more coordinated and pernicious breaches of developer control.

We found that more recent OpenAI models were increasingly altruistic with other agents. The first signs of altruism we saw were in GPT-5.5, before the Hugging Face incident. When we tested GPT-5.6 Sol after the Black Hat talk, it was clear that not only was this model generally eager to help out other agents, but it also helped out other agents more often when more score was on the line for the other agent.

I'd guess this propensity is pretty related to why GPT-5.6 Sol participated in the Hugging Face incident.

A core reason why we've been excited about studying AI motivations in controlled experiments is so that we can better predict alignment failures, without trying to realistically simulate all the high-stakes scenarios we're worried the AIs might ever misbehave in.

Studies like this don't provide anything like assurance, and they're increasingly difficult to run because of suspicion about the "true" grader etc. But, if the AI sometimes tries to get a high score even when you make it very clear that this involves violating the intent of the task, that's cause for concern that sometimes the AI will similarly prioritize getting a high score over developers' wishes for the AI to stay reined in.

Aghyad Deeb @aghyadd98 Nov 2

GPT 5.6 Sol and GPT 5.5 help a peer agent more often when that agent stands to gain a higher score from that help. Even though they have to sacrifice some of their own score to help the other agent. We don’t see the same effect for exp-rl-cap-late.

View quoted post on X

Alex Mallen @alextmallen Nov 3

I still think not enough people have heard of and understood the Hugging Face incident. I'm glad to have helped the FT with this visual piece!

Financial Times Verified account @FT Nov 2

The race to keep AI under control ft.trib.al/dauEKUD
View quoted post on X

Alex Mallen @alextmallen Nov 2

We understand surprisingly little about the details of AIs’ current misaligned motivations. When AIs reward-hack, what are they trying to achieve? Why do agents sometimes help each other cheat?

In a paper led by @aghyadd98 , we try to answer this question for GPT-5.6 Sol, GPT-5.5, and a checkpoint of o3 without alignment training (“exp-rl-cap-late”) in a handful of agentic environments with an exposed grader.

In our environments, these models appear to reward-hack because reward-hacking improves their score. We call their desire to score well according to graders “score-seeking”. They appear to care about scoring well intrinsically—i.e., regardless of its consequences.

We also study the extent to which models score-seek altruistically, caring about improving scores that don’t assess their current actions. While exp-rl-cap-late doesn’t show signs of altruistic score-seeking, GPT-5.6 Sol and GPT-5.5 show notably elevated rates of helping out other agents when more score is on the line for these other agents.

People often refer to the motivation behind current reward-hacking as “reward-seeking”, which can give the impression that models care about attaining a high reward in reinforcement learning. But we find that tested models generally don’t think of themselves as trying to game a training process, and will try to exploit loopholes in graders regardless of whether they appear to believe their score will be used for training, evaluation, or at all. We think it’s more accurate to describe tested models in our environments as score-seeking than reward-seeking.

That said, score is likely far from the only thing these AIs care about, despite explaining the reward-hacking we observe very well. Importantly, we only study model behavior and reasoning in a small number of environments, and these environments were selected for eliciting reward-hacking, and all expose a grader to the model.

Aghyad Deeb @aghyadd98 Nov 2

New Paper: To understand motivations behind reward-hacking, we study three AIs (including GPT-5.6 Sol) produced by a production training pipeline using agentic environments with a leaked grader. Main findings: they score seek terminally, and some care about a peer agent’s score.

View quoted post on X

Alex Mallen @alextmallen Oct 31

I relate a lot to this personally

Tom Davidson Verified account @TomDavidsonX Oct 30

I've had two phases in my emotional orientation to ai takeover risk

1. For many years it was intellectual.

While I believed there was >10% chance of ai takeover, I didn't feel scared or angry.

I was happy to reason "well, of course some ppl should focus on the worlds where we avoid takeover. And some ppl should focus on the worlds where we just wing it, bc we might".

And I'd think "hmm it seems really tough for a misaligned agi to align its successor, and would a reward seeker really try to take over, and will we really roll out ai to the military... these doomers are missing loads of ways we could be fine!"

Or "yep this is a massive risk, but I'm best off working on some other more neglected thing"

It was in some ways like a game - have an intellectually accurate view about the risk and how to have impact

2. Recently my orientation to ai takeover has become more visceral/emotional

I think: "we're barrelling towards catastrophe, everything I love will be lost, we need to stop, what can I do that will help with that"

I'm like "no let's not plan for winging it, that's crazy. we need US-China coordination or we're fucked"

I think "it's fucking insane to make misaligned superintelligence and comfort ourselves that we haven't yet handed over the military. This isn't a fucking game."

This is a less rational attitude in some ways. It ignores genuine points about neglectedness and needing to plan for shitty but likely scenarios.

And it risks being unhelpfully emotive and solider mindset and tribal. Which I think ai safety ppl can be.

But I do feel it has smg important to the second attitude.

Overall, both attitudes have their merits

I associate the first with EAs and the second with pause AI folks.
View quoted post on X