Kenny Workman

A talk on verifiable environments for agents in biology

0:00 Where does experimental data come from?
1:41 Modern bio research organized around measurement
2:44 Data analysis is an executable substrate for science
3:23 Five years of learnings building products for pharma
4:13 Trying to use coding models for biology workflows
5:40 Why frontier models can't be trusted (yet)
6:44 Sequencing based spatial analysis
7:38 SpatialBench design principles
8:26 Anatomy of a single evaluation
9:40 Human verification and early long horizon extensions
13:12 Grading functions and the case against rubrics
14:07 New work in multi-omics, therapeutics, biosecurity

Rare diseases collectively affect an estimated 300 million people worldwide. For patients and families, reaching a diagnosis can take years of referrals and testing. One place AI agents could help is in interpreting each new sequencing result or functional assay alongside the patient’s symptoms, family history, and earlier findings.

We introduce RareDiseaseBench, a benchmark of 35 diagnostic workups reconstructed from published rare disease cases. Each agent starts with the patient’s presentation and part of the medical record. Over one or more visits, it must decide which tests to request, explain why, analyze the files it receives, and propose a diagnosis and management plan.

Five cases from the benchmark:

- A 13-year-old boy has growth restriction, hearing loss, and severe chilblains. An MN1 variant explains several developmental findings. Long-read sequencing reveals a translocation disrupting SAMHD1 that accounts for his inflammatory symptoms.

- A 21-year-old man has heterotaxy with right atrial isomerism. A variant just beyond the standard ZIC3 transcript falls within an intron of an alternative transcript, causing 57 extra bases to be spliced into the RNA.

- A boy develops progressive scalp hair loss while his eyebrows and body hair are preserved. A scalp biopsy shows miniaturized follicles, and sequencing reveals a 357-kb duplication spanning TAB3 and neighboring genes.

- A 16-year-old boy with typical growth has a language disorder, learning disability, and autism. He carries two inherited variants in RNU2-2, a small nuclear RNA gene outside the coverage of his exome sequencing.

- A boy with delayed motor milestones and enlarged calves has a creatine kinase level of 10,100 U/L. At a second visit, an outside lab reports a DMD inversion. Targeted long reads show duplicated segments containing exons 1 and 2 joined head to head.

Each configuration made three attempts on each of the 35 cases. Gemini 4 Argon with Pi passed 46 of 105 attempts (43.8%). Claude Opus 5.5 with Claude Code and GPT-6 Astra with Pi each passed 35 of 105 (33.3%). A pass required the diagnosis and management plan to meet every criterion in the case-specific rubric. Timeouts counted as failures.

Among the 14 models that passed at least one case, 2,097 runs failed. Data acquisition and diagnostic workup was the most frequent primary failure category (36%). In 15% of all analyzed failures, the agent failed to pursue the type of data the case required. In another 12%, it omitted a necessary non-sequencing investigation, such as a tissue stain, imaging study, or laboratory test.

Paper: latch.bio/raredisease
Leaderboard: benchmarks.bio/raredisease
Blog: blog.latch.bio/p/rarediseaseb…

We think rare disease diagnosis is one of the areas where AI systems have the most potential to make rapid progress and directly benefit patients over the next few years.

Incredible work from @chriswzou and team.

@arjunomics documented very similar behavior in Kimi-K3 back in July

x.com/arjunomics/sta…

These are always very high quality and super technical. Always take away something valuable and meet cool people.

We hosted one with Shreya at our office last year, where we presented a distributed filesystem built on Postgres + S3 for workloads in biology.

x.com/kenbwork/statu…

benchmarks.bio/respvirus-surv… includes LSVP results