Handshake (@joinHandshake)

Pinned post

Handshake @joinHandshake Jun 10, 2025

Introducing Handshake AI—the most ambitious chapter in our story. We leverage the scale of the largest early career network to source, train, and manage domain experts who test and challenge frontier models to failure for the top AI labs.

1:35

Handshake @joinHandshake 1d

Pretty great progress through @reflection_ai on the development of their alignment teacher.

It’s nuanced across domains to know when an agent should refuse a request. For example, if someone asks how to create a protein, it could be used for drug discovery or to create a bioweapon.

They built a robust eval set aligned to their policies. They improved against those policies by reducing over-refusals by 48.7%, while only decreasing refusals of valid requests by 3.1%.

We're proud to have worked with them on the eval design and adversarial probing.

Brandon Amos Verified account @brandondamos 1d

We've posted a 156-page tech report on Beam! We have good systems and it's worth reading. It covers all phases through pre-training, mid-training, SFT, RL, and OPD and goes into the infra, data, and scientific decisions we made throughout the process

📚 reflection.ai/documents/beam…

View quoted post on X

Handshake @joinHandshake 1d

⏳ 30 Days to #NeurIPS202

The Handshake AI research team will be presenting two papers at NeurIPS 2026 Sydney.

BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows
📄 Preprint: arxiv.org/pdf/2604.11304

Verifying Agents in Rubric-Graded Environments
⚙️ Code: github.com/Handshake-AI-R…

Authored by Elaine Lau, @awws0me, Ronak Chaudhary, Vaibhav Kumar, @andreas_plesner, @jomulr, and the rest of the team.

Can’t wait to see you there!

Handshake @joinHandshake 1d

@ArtificialAnlys just released Handshake's VIALS benchmark!

VIALS evaluates models on their ability to perform visual reasoning tasks in life sciences and biotech. Think: a biologist in a lab examining a molecule to determine its structure.

Gemini 4 Carbon leads at 64%, followed closely by GPT-6 Astra at 63% and Claude Opus 5.5 at 61%.

We believe AI offers great promise to assist with drug discovery and medicine, and hope the VIALS benchmark will serve that end.

Artificial Analysis Verified account @ArtificialAnlys 1d

We’re launching VIALS-AA, a leaderboard for Handshake AI’s VIALS benchmark for visual reasoning in life sciences and biotech research

Visual artifacts are central to how scientists analyze experiments and make research decisions, but interpreting them correctly remains a challenge for frontier models. VIALS’ 161 tasks span eight domains and need models to reason over artifacts including gel blots, flow cytometry plots, plasmid maps, and phylogenies. Each task requires models to interpret an artifact and draw a specific scientific conclusion.

We evaluate models by running each task three times in Stirrup, our open source agent harness, with code execution and image inspection tools. GPT-6 Luna (medium) grades each answer against the reference answer, and we report the share of tasks judged correct in all three repeats (pass^3).

Key takeaways:

➤ The top three models are separated by 3 points: Gemini 4 Carbon (high) from @GoogleDeepMind leads narrowly with 64%, followed by GPT-6 Astra (max) from @OpenAI at 63% and Claude Opus 5.5 (max with fallback) from @AnthropicAI at 61%

➤ Astra and Sol achieve similar scores to Opus and Sonnet at a much lower Cost per Task: Claude Opus 5.5 (max with fallback) averages $1.64 per task, ~29× as much as Sol (~$0.056). This price difference is significantly driven by turn use - GPT-6 Astra (max) and GPT-6.1 Sol (max) average ~8 turns per task, compared with ~27 for Claude Opus 5.5 (max with fallback) and ~25 for Claude Sonnet 5.5 (max with fallback)

➤ Open weights models trail the highest score by ~19 points: Kimi K3.1 Pro leads open weights models on VIALS-AA at 45%, narrowly ahead of MiMo-V2.6-Pro at 43%

➤ 14 of 161 tasks saw safety refusals on at least one model: Claude Sonnet 5.5 (max with fallback) refused on 6 tasks, the most of any model, and half of the 14 affected tasks saw refusals from multiple models

Thank you to @joinHandshakeAI and the VIALS authors for creating the benchmark and collaborating with us on bringing it to Artificial Analysis!

View quoted post on X

Handshake @joinHandshake 2d

Congratulations to the team at @thinkymachines for their release of Inkling 2 Small! We’re excited to see what’s next. 👀

Thinking Machines Verified account @thinkymachines 2d

Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.

Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.

thinkingmachines.ai/news/inkling-2…
View quoted post on X

Handshake

Latest posts 5