Pinned post
Can agents use experimental feedback to establish new material capabilities that hardware teams can reliably design around?
Today we’re releasing SDL-2, an open evaluation of how agents choose experiments and decide whether a material process will reliably meet a requirement.
We’re interested in the full process of scientific investigation. That means discovering which relationships matter, choosing experiments that distinguish competing explanations, revising an explanation when results surprise and knowing when the evidence is strong enough to act.
The experiments run in a digital twin of magnet processing, calibrated on measurements from our campaign with Rensselaer Polytechnic Institute (@rpi). Agents investigate unfamiliar powders with an uncertain physics model, a limited experimental budget and results that vary from batch to batch.
Across 32 simulated worlds, frontier models reached correct decisions supported by their own measurements in 44–50% of investigations, compared with 25–28% for other language models and classical experiment design. One in three of their correct answers were unsupported.
Our larger ambition is for materials development to keep pace with hardware design, so teams can design around the capabilities they request, not only those that already exist. SDL-2 makes one part of that measurable, how agents investigate.
We're releasing the evaluation code, harness, digital twin and all 256 investigation traces.
Technical Report: dynamicalsystems.ai/blog/sdl-2
Code: github.com/Dynamical-Syst…
Dataset: huggingface.co/datasets/Dynam…
Today we’re releasing SDL-2, an open evaluation of how agents choose experiments and decide whether a material process will reliably meet a requirement.
We’re interested in the full process of scientific investigation. That means discovering which relationships matter, choosing experiments that distinguish competing explanations, revising an explanation when results surprise and knowing when the evidence is strong enough to act.
The experiments run in a digital twin of magnet processing, calibrated on measurements from our campaign with Rensselaer Polytechnic Institute (@rpi). Agents investigate unfamiliar powders with an uncertain physics model, a limited experimental budget and results that vary from batch to batch.
Across 32 simulated worlds, frontier models reached correct decisions supported by their own measurements in 44–50% of investigations, compared with 25–28% for other language models and classical experiment design. One in three of their correct answers were unsupported.
Our larger ambition is for materials development to keep pace with hardware design, so teams can design around the capabilities they request, not only those that already exist. SDL-2 makes one part of that measurable, how agents investigate.
We're releasing the evaluation code, harness, digital twin and all 256 investigation traces.
Technical Report: dynamicalsystems.ai/blog/sdl-2
Code: github.com/Dynamical-Syst…
Dataset: huggingface.co/datasets/Dynam…
0:55