Vision evaluations need a lot of work.
Announcing our recently accepted NeurIPS 2026 workshop paper investigating the problems with the state of video evals.
The paper also proposes a more faithful way to evaluate vision for agents.
Announcing our recently accepted NeurIPS 2026 workshop paper investigating the problems with the state of video evals.
The paper also proposes a more faithful way to evaluate vision for agents.
Do video benchmarks actually test video understanding? Excited to announce our NeurIPS 2026 workshop paper investigating problems with the state of video evals, and ways to address them.View quoted post on X
One of the many problems is highlighted in the image below (option-conditioned blind solvability)