AVERI

AI companies make safety calls based in large part on internal tests they develop themselves. It’s rare for there to be independent assessment of the quality of those tests.

Earlier this year, @Amazon and @AVERIorg agreed to a pilot project in which AVERI assessed the quality of Amazon’s internal safety evaluations. Today we’re releasing the result of that independent review. As a result of this project, Amazon is changing how it tests future models.

Our review of Amazon’s internal AI safety evaluations confirmed several strengths such as topical diversity, and we found the evaluations to be consistent with Amazon’s public statements on them. At the same time, we found areas for improvement:

- On attack sophistication, some prompts represented relatively straightforward attack vectors that would test safety filters rather than deeper model reasoning.
- On evaluation signal quality, we identified a concern that superficial cues, rather than safety reasoning, may drive some model refusals, so passing these evaluations might not indicate robust safety behavior in novel situations.
- On linguistic and stylistic diversity, some multilingual prompts were included, but prompts across risk categories were predominantly in English, and some prompt structures were reused across scenarios.
- On evaluation freshness, a small number of prompts contained temporal references that could become outdated.

In response, Amazon reports four changes for future internal evaluations:

- Refreshing its test sets to remove redundant prompts and add more challenging adversarial scenarios;
- Introducing a tiered difficulty calibration framework;
- Expanding multilingual coverage across all risk categories; and
- Establishing a quarterly review schedule to identify staleness and incorporate emerging attack patterns.

Why does this matter? A passing grade on an easy test doesn’t tell us much on its own. As the stakes of AI development and deployment grow, we think that this kind of independent evaluation of internal evaluations (“metaevaluation”) will grow in importance.

We appreciate Amazon opening their internal evaluations up to outside review. We hope more companies take this step to help make their models safer.

While no clear standards exist for metaevaluation yet, @AVERIorg is working to change that. We look forward to sharing more pilot results in the coming months and will continue our work developing standards with institutions like @aievalforum.

Read AVERI’s full blog post here: averi.org/ourwork/averi-…

Read Amazon’s blog post on the collaboration here: amazon.science/blog/strengthe…

AVERI is glad to see that independent auditing of frontier AI was prominently featured in the White House accord signed today.

The details of such auditing matter greatly, which is why AVERI works with other members of the @aievalforum to identify best practices and develop standards in areas such as auditor independence, minimum access requirements, and audit scope.

We believe that audits conforming to such standards should ultimately be required for leading AI companies, though voluntary commitments are a step forward.

AVERI is proud to be a member of AEF and looks forward to continued work on establishing clear standards as the evaluation ecosystem matures.

AVERI's Executive Director, @Miles_Brundage, spoke with Business Insider about recent discussions of embedded auditing:

AVERI commends recent statements by the leaders of Anthropic, OpenAI, and SpaceX regarding the importance of embedding independent experts within frontier AI companies. We are committed to working with various stakeholders to advance standards for frontier AI auditing, and moving towards binding requirements for auditing that aligns with those standards.

Our flagship paper, “Frontier AI Auditing,” published in collaboration with experts in industry, civil society, and academia, lays out a vision for third-party auditing of frontier AI companies’ systems and practices and includes embedding as a key component. Embedding helps facilitate deep, secure access to non-public information.

Moving embedded auditing from a voluntary practice to an effective and universal industry requirement requires not only pilot projects, but also standards, tooling, and thoughtful policy design. AVERI exists to provide these, and we are conducting pilot projects aimed at informing standards for different aspects of AI audits.

Embedded auditing will require clear standards for who counts as a qualified and independent auditor, how audits should be conducted and reported, and what safety and security standards companies should be audited against. Fortunately, we are not starting from scratch: with evaluators across the ecosystem, we coauthored a standard (AEF-1) which covers, e.g., conflict of interest disclosure and management in an AI context.

Learn more about vision for frontier AI auditing here: averi.org/ourwork/fronti…