Sören Mindermann

Marcus Williams Verified account @Marcus_J_W · 5d

🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.

To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.

View quoted post on X

Toby Ord Verified account @tobyordoxford · Oct 21

A recent set of posts by @ramez on the current state of recursive self improvement highlighted that OpenAI's report on their own research automation shows that models struggle to do their R&D work — the models' 80% time horizon is only 15 minutes:
🧵

View quoted post on X

Arvind Narayanan Verified account @random_walker · Oct 15

There is widespread frustration about the AI industry’s “automate everything" agenda. But what would it take to steer AI toward augmenting human capabilities instead? In a new essay, Gabriel Unger (@_gabrielunger) and I lay out a research agenda and plan for action. normaltech.ai/p/steering-ai-…

Our thesis is that the reason why AI-for-automation currently dominates isn’t just economic incentives but also deeper cultural issues in computer science as a field. And that can be changed.

- We sketch four important research directions that are needed in order to build collaborative or augmentative AI. The automation paradigm largely ignores these technical challenges.
- We advocate for Human-Computer Interaction and Human-Centered AI to be elevated as a subfield in status, influence, and funding, and changing AI curricula to emphasize the role of the human and decrease the centrality of the pure-automation problem framing.
- We point out that there are a surprising number of efforts in economics, computer science, industry, and policy all trying to bring about a shift toward augmentation, but not enough collective awareness. Better connecting these movements would amplify their power.
- We think the automation agenda is struggling on its own terms. Compiling evidence of the limits of general-purpose agents for automation and loudly making the case will encourage more companies to take collaboration more seriously as a goal.
- We predict that once more workers experience this way of working and recognize that there are alternatives to the delegate-and-supervise pattern, it will generate more demand, creating a flywheel effect.
- Targeted policy levers can play a small but important role in amplifying both the supply and demand for pro-worker AI. We don’t need clumsy top-down mandates.

We're not against automation. But we think the goal of AI must be broader.

This is the start of a long-term project. I know there are far more like-minded efforts out there than we’re aware of. I’d love to hear about some of them.

View quoted post on X

Dylan HadfieldMenell Verified account @dhadfieldmenell · Oct 6

This is one of the easiest wins on AI regulation. Whenever AI does something that would be a tort or crime for a human, we should be able to hold *some* person responsible --- the user is responsible for intended behavior and the developer is responsible for unintended behavior.

View quoted post on X