Pinned post
Interesting. (Founder of Z.ai, creator of GLM AI models.)View quoted post on X
Aaron Scher
Verified account This account is verified. Learn more@aaronscher
Technical AI Governance research @MIRIBerkeley Speaking for myself and in a personal capacity
833 Following 1.1K Followers 1.5K Posts
Latest posts 5
Extremely concerning. This report updates me toward the 2x AI R&D uplift threshold being very close to achieved or already achieved, making full RSI in 2027 likely. Nobody has an alignment plan.
We need to back off. If anyone builds it, everyone dies—apparently in <2 years.
We need to back off. If anyone builds it, everyone dies—apparently in <2 years.
1. Benchmarks provide one piece of evidence for acceleration. Since Oct 2025, AECI scores for Anthropic’s models have grown at 1.9x [95%: 1.5–2.8] the earlier rate. Analysis choices affect this estimate, and it remains unclear how benchmark scores map to real-world capabilities.View quoted post on X
I dislike the term "compute campus". Campuses are where our kids go to school. They are a deeply human thing. Machines are not human.
It is true that AIs may at some point be worthy of moral concern; we should not torture them as we do not torture dogs. But they are not human.
It is true that AIs may at some point be worthy of moral concern; we should not torture them as we do not torture dogs. But they are not human.
OpenAI kicked off its big charm offensive this morning after data center moratoriums went 43-1 during the midterms, and many candidates rode a wave of public opposition to the industry to huge victories.
Data centers have been rebranded as Compute Campuses. OpenAI has also released a National Community Compact and a Compute Campus Policy Blueprint.
OpenAI’s National Community Compact: Building With Communities, Not Just In Them:
cdn.openai.com/pdf/community-…
AI Infrastructure That Works for America: Our Compute Campus Policy Blueprint:
cdn.openai.com/pdf/compute-ca…
The most significant change is probably no more NDAs, from page 7:View quoted post on X
Thanks to TVW's The Impact for having me on the show!
We covered topics like:
- What are the main AI threats? (human extinction) Are they a distraction from other AI harms? (no)
- Does giving up ASI mean giving up all AI benefits? (no)
And much more!
tvw.org/video/the-impa…
We covered topics like:
- What are the main AI threats? (human extinction) Are they a distraction from other AI harms? (no)
- Does giving up ASI mean giving up all AI benefits? (no)
And much more!
tvw.org/video/the-impa…
tvw.org The Impact - The AI Regulation Debate - TVW This week, we explore state and national discussions surrounding AI regulation. The episode covers recent legislation and reported AI-testing incidents that Some quotes from the full post:
- "While we were ultimately not able to pinpoint the exact cause of this behavior, we identified that it seems to increase during RL. We additionally share two initial attempts at root-causing investigations we performed respectively on an RL grader and an SFT dataset, despite them not leading to a conclusive explanation."
- The behavior was noticed "Shortly after internal deployment of GPT-6 Astra began", which likely means August or earlier this year, given the Sep 3 public release of the model. Concerning that it took so long for this research to happen and be published, and that it's still quite inconclusive about the causes of this concerning behavior.
- "In later widescale analyses we found cases of verbalized underperformance. Our monitoring system did not flag these trajectories as severe enough misalignment to trigger human review."
- Model CoT: "Need refrain overly announcing refusal."
- "Astra completed about 16 percent of hidden rubric items under the capability framing and only 10 percent under the safety framing. This is a gap of about 6 percentage points and roughly 40 percent fewer completed rubric items." Small percentage point difference but huge relative difference!
- "Adding instructions to be careful to the developer message increased safety-task underperformance by 50 percent. This points toward an overly cautious response to safety research as the simplest current explanation, although we cannot rule out strategic motives that Astra did not externalize in CoT."
- "While we were ultimately not able to pinpoint the exact cause of this behavior, we identified that it seems to increase during RL. We additionally share two initial attempts at root-causing investigations we performed respectively on an RL grader and an SFT dataset, despite them not leading to a conclusive explanation."
- The behavior was noticed "Shortly after internal deployment of GPT-6 Astra began", which likely means August or earlier this year, given the Sep 3 public release of the model. Concerning that it took so long for this research to happen and be published, and that it's still quite inconclusive about the causes of this concerning behavior.
- "In later widescale analyses we found cases of verbalized underperformance. Our monitoring system did not flag these trajectories as severe enough misalignment to trigger human review."
- Model CoT: "Need refrain overly announcing refusal."
- "Astra completed about 16 percent of hidden rubric items under the capability framing and only 10 percent under the safety framing. This is a gap of about 6 percentage points and roughly 40 percent fewer completed rubric items." Small percentage point difference but huge relative difference!
- "Adding instructions to be careful to the developer message increased safety-task underperformance by 50 percent. This points toward an overly cautious response to safety research as the simplest current explanation, although we cannot rule out strategic motives that Astra did not externalize in CoT."
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.View quoted post on X
