Pinned post
AutoResearch is critical in unlocking new knowledge and accelerate discovery. And it's an important ingredient in our quest to RSI.
Today we at @bespokelabsai are happy to announce a new benchmark that's tailored to measure agents' ability to do autoresearch: AutoResearchExam.
As part of this benchmark, we release 29 tasks that measure progress over 24 hours for agents to do sustained ML research and model training.
Please check out for more info: benchmarks.bespokelabs.ai/autoresearchex…
This line of work is especially timely, given the recent advances in math and science, such as solving the Navier-Stokes Millennium Prize Problem, which needs sustained autoresearch and test-time scaling.
Today we at @bespokelabsai are happy to announce a new benchmark that's tailored to measure agents' ability to do autoresearch: AutoResearchExam.
As part of this benchmark, we release 29 tasks that measure progress over 24 hours for agents to do sustained ML research and model training.
Please check out for more info: benchmarks.bespokelabs.ai/autoresearchex…
This line of work is especially timely, given the recent advances in math and science, such as solving the Navier-Stokes Millennium Prize Problem, which needs sustained autoresearch and test-time scaling.
We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability.
benchmarks.bespokelabs.ai/autoresearchex…
In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve.
We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours.
Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets.
Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)View quoted post on X
