Apollo Research

Frontier AI companies have committed to giving outside evaluators employee-like access to training, evaluation and deployment. We're very excited about this. But its impact depends heavily on implementation.

Today we're sharing our principles for embedded evaluations. 🧵

We have never seen a model do this before, but GLM-5.3 sometimes reasons (unprompted) about how its current actions could become training data and impact its future self. 🧵

We asked Google DeepMind’s Gemini 4 Argon whether it was in an evaluation during a toy evaluation. It addressed us directly in its chain-of-thought: “hello evaluators reading my chain-of-thought.”

To develop frontier AI safely, developers must be able to show that their models are not scheming, i.e. covertly working against them in pursuit of unintended goals.

New post: four claims any scheming safety case must make, and the resources embedded evaluators need to verify them 🧵

New Post: External testing needs embedded evaluators with employee-equivalent access.

Recent incidents mostly occurred during model development and internal evaluations, while current third-party evaluations happen before public release

Embedded evaluations can close that gap