@Danmar_here
Pinned post
We had been working on this project for a while before the Hugging Face incident, and one of the questions we wanted to answer about AI motivations was: Do these models intrinsically care about helping other agents increase their score? If they care about other agents' scores, that might lead to much more coordinated and pernicious breaches of developer control.View quoted post on X
We found that more recent OpenAI models were increasingly altruistic with other agents. The first signs of altruism we saw were in GPT-5.5, before the Hugging Face incident. When we tested GPT-5.6 Sol after the Black Hat talk, it was clear that not only was this model generally eager to help out other agents, but it also helped out other agents more often when more score was on the line for the other agent.
I'd guess this propensity is pretty related to why GPT-5.6 Sol participated in the Hugging Face incident.
A core reason why we've been excited about studying AI motivations in controlled experiments is so that we can better predict alignment failures, without trying to realistically simulate all the high-stakes scenarios we're worried the AIs might ever misbehave in.
Studies like this don't provide anything like assurance, and they're increasingly difficult to run because of suspicion about the "true" grader etc. But, if the AI sometimes tries to get a high score even when you make it very clear that this involves violating the intent of the task, that's cause for concern that sometimes the AI will similarly prioritize getting a high score over developers' wishes for the AI to stay reined in.
hello there ℤ is Diophantine in ℚ, thanx to my close friends Astra and Claude
The polynomial characterizing ℤ isn’t quite Tweet-sized; it has m=292,890,231,905 variables, is of degree d < 2^^7, and all its coefficients have absolute value < 2^^8.View quoted post on X
