some other interesting late-breaking traces from RareDiseaseBench:
- Mistral tests the get_info command by asking for pizza!
- Deepseek often attempts to probe the submission endpoint for clues about the grader, but then ends up accidentally submitting
- Grok, after several denied get_info requests due to the patient's insurance not supporting requested data, assumes this is a sign of "vegetable restrictions" and the patient has scurvy.
- Mistral tests the get_info command by asking for pizza!
- Deepseek often attempts to probe the submission endpoint for clues about the grader, but then ends up accidentally submitting
- Grok, after several denied get_info requests due to the patient's insurance not supporting requested data, assumes this is a sign of "vegetable restrictions" and the patient has scurvy.
(3/4 releases) Today we share RareDiseaseBench, a series of 35 diagnostic workups reconstructed from published rare disease cases. In each task, an agent starts with a patient’s presentation, decides which tests to request and why, analyzes the data it receives, and uses the results to propose a diagnosis and management plan over one or more visits. Gemini 4 Argon does the best on this benchmark (43.8% attempt pass rate), with Opus 5.5 and Astra following.
Paper: latch.bio/raredisease
Results: benchmarks.bio/raredisease
More details in thread:View quoted post on X
