We built an evaluation platform that now runs millions of evals every week.
You might assume the hardest part was technical: scale, reliability, or building the right infrastructure.
Those were certainly hard. But one of my biggest learnings was that the deeper question is often not:
“Can we run this eval?”
It’s:
“Are we evaluating the right thing?”
Engineering cannot answer that alone.
A product manager understands what success looks like for the customer. A domain expert can spot an answer that sounds plausible but is actually wrong. Design and research bring the user’s perspective. Engineering turns that collective judgment into something measurable and repeatable.
In many cases, the most valuable eval starts as a conversation and not code.
AI Evals aren’t simply an engineering harness or a final check before shipping.
They’re a shared language that helps cross-functional teams agree on what “good” looks like and determine whether the product is actually getting better.
In this @aiDotEngineer talk, @swaroopch and I share some of what we learned while building eval infrastructure at scale and why successful AI evaluation has to be a team sport.
youtu.be/bMjlRrWjdT0?is…
You might assume the hardest part was technical: scale, reliability, or building the right infrastructure.
Those were certainly hard. But one of my biggest learnings was that the deeper question is often not:
“Can we run this eval?”
It’s:
“Are we evaluating the right thing?”
Engineering cannot answer that alone.
A product manager understands what success looks like for the customer. A domain expert can spot an answer that sounds plausible but is actually wrong. Design and research bring the user’s perspective. Engineering turns that collective judgment into something measurable and repeatable.
In many cases, the most valuable eval starts as a conversation and not code.
AI Evals aren’t simply an engineering harness or a final check before shipping.
They’re a shared language that helps cross-functional teams agree on what “good” looks like and determine whether the product is actually getting better.
In this @aiDotEngineer talk, @swaroopch and I share some of what we learned while building eval infrastructure at scale and why successful AI evaluation has to be a team sport.
youtu.be/bMjlRrWjdT0?is…
www.youtube.com AI Evals for Cross-Functional Teams — Nachiket Paranjape & Swaroop Chitlur Haridas, DoorDash The people annotating DoorDash's eval data are not engineers, and t... 
