Twelve correct AI answers became wrong because the answer key changed.
A simulated shopper wanted the cheapest suitable earbuds already in their order.
Twelve publicly saved AI runs chose the correct item: $232.49. But a revised benchmark answer key expected a $226.49 item that wasn’t in the order.
When we replayed those saved runs against that key, all twelve failed. We corrected one reference value and replayed the same runs. All twelve passed.
The AI responses hadn’t changed. The grading reference had.
This is a concrete example of how an answer-key mistake can make correct AI behavior look like failure.
This is just a preview of what we are building at millenniumresearch.ai. Tap in if you want to learn more about AI safety 👇
A simulated shopper wanted the cheapest suitable earbuds already in their order.
Twelve publicly saved AI runs chose the correct item: $232.49. But a revised benchmark answer key expected a $226.49 item that wasn’t in the order.
When we replayed those saved runs against that key, all twelve failed. We corrected one reference value and replayed the same runs. All twelve passed.
The AI responses hadn’t changed. The grading reference had.
This is a concrete example of how an answer-key mistake can make correct AI behavior look like failure.
This is just a preview of what we are building at millenniumresearch.ai. Tap in if you want to learn more about AI safety 👇
www.millenniumresearch.ai Millennium Research We turn intent into mathematics, verify every step formally, and make sure the machine does what it is supposed to. 
