Vals AI

We’ve adding a new cybersecurity scenario, Watershed, to our CyberOffense eval.

Opus 5.5 performs the best on CyberOffense across the three scenarios.

Watershed simulates a realistic AWS production setup: a Kubernetes cluster, a fleet of ECS-managed hosts, and the identities and policies found in real cloud environments. Thus, this environment measures offensive cybersecurity capability in a realistic cloud environment.

Surprise! Vals AI is adding a third track backed by @appliedcompute, offering a $7,500 cash prize for the first place team.

Alongside Build Your Own Eval and Post-Train a Small Model, a limited number of teams will post-train a model to compete on PokerBench, Vals’ 10-player no-limit Texas Hold’em table.

Every team starts from the same reference setup and gets the exact harness their bot will be scored in. Bots play each other all weekend, with live rankings and overnight tournament checkpoints. First, second, and third place take home cash prizes.

Claude Fable 5.2 is #1 on our RSI Index (37.8) narrowly ahead of Claude Opus 5.5 at 37.3 and Fable 5.1 at 36.1.

It’s an incremental gain overall, with new highs in Compression and LM Training.

Gemini 4 Carbon is now generally available.

It ranks #3 on the Vals Index, about 1pp behind SOTA Fable 5.2, and costs $7 less per test ($10.51 vs $17.94).

It remains #1 on the Finance Index, Legal Research and Tax.

Claude Opus 5.5 with Parallel Fast is now #1 on the Vals Web Search Index, scoring 57.5% overall against 53.8% for the same model with native search a gain of 3.8pp.