Pinned post
That’s the idea - train the humans before the robots do.
But I thought AI would take all jobs ?View quoted post on X
Garrett Lord
Verified account This account is verified. Learn more@GarrettLord
CEO/Founder of @joinhandshake
2.7K Following 7.5K Followers 3K Posts
Latest posts 5
In drug discovery and disease research, scientists visually interpret hundreds of experimental results to guide critical decisions.
VIALS evaluates how accurately AI can interpret these same types of scientific images (assays), spanning the types encountered in biotech/pharma workflows.
Current models struggle, lacking scientific knowledge and visual perception, particularly when applying scientific reasoning to images. In contrast, most scientists with the appropriate PhD easily do these tasks in minutes. Until AI can do the same, it usefulness in industry life sciences workflows will remain limited.
VIALS evaluates how accurately AI can interpret these same types of scientific images (assays), spanning the types encountered in biotech/pharma workflows.
Current models struggle, lacking scientific knowledge and visual perception, particularly when applying scientific reasoning to images. In contrast, most scientists with the appropriate PhD easily do these tasks in minutes. Until AI can do the same, it usefulness in industry life sciences workflows will remain limited.
We’re launching VIALS-AA, a leaderboard for Handshake AI’s VIALS benchmark for visual reasoning in life sciences and biotech research
Visual artifacts are central to how scientists analyze experiments and make research decisions, but interpreting them correctly remains a challenge for frontier models. VIALS’ 161 tasks span eight domains and need models to reason over artifacts including gel blots, flow cytometry plots, plasmid maps, and phylogenies. Each task requires models to interpret an artifact and draw a specific scientific conclusion.
We evaluate models by running each task three times in Stirrup, our open source agent harness, with code execution and image inspection tools. GPT-6 Luna (medium) grades each answer against the reference answer, and we report the share of tasks judged correct in all three repeats (pass^3).
Key takeaways:
➤ The top three models are separated by 3 points: Gemini 4 Carbon (high) from @GoogleDeepMind leads narrowly with 64%, followed by GPT-6 Astra (max) from @OpenAI at 63% and Claude Opus 5.5 (max with fallback) from @AnthropicAI at 61%
➤ Astra and Sol achieve similar scores to Opus and Sonnet at a much lower Cost per Task: Claude Opus 5.5 (max with fallback) averages $1.64 per task, ~29× as much as Sol (~$0.056). This price difference is significantly driven by turn use - GPT-6 Astra (max) and GPT-6.1 Sol (max) average ~8 turns per task, compared with ~27 for Claude Opus 5.5 (max with fallback) and ~25 for Claude Sonnet 5.5 (max with fallback)
➤ Open weights models trail the highest score by ~19 points: Kimi K3.1 Pro leads open weights models on VIALS-AA at 45%, narrowly ahead of MiMo-V2.6-Pro at 43%
➤ 14 of 161 tasks saw safety refusals on at least one model: Claude Sonnet 5.5 (max with fallback) refused on 6 tasks, the most of any model, and half of the 14 affected tasks saw refusals from multiple models
Thank you to @joinHandshakeAI and the VIALS authors for creating the benchmark and collaborating with us on bringing it to Artificial Analysis!View quoted post on X
Best birthday present ever. Tatum Lord, our Halloween baby!
Welcome to the world, Tatum ❤️
@jessicarwolf
Welcome to the world, Tatum ❤️
@jessicarwolf
Data is the bottleneck.
Brett Adcock says Figure is now printing 81 minutes of data every second, and every five days it prints an entire waking human lifetime. He also said Figure spent an estimated $8.1 million on data last week.View quoted post on X
congrats @AndrewDai!
Today we're thrilled to announce the release of our first frontier visual reasoning modelView quoted post on X
