Generating good and useful rubrics is hard. Nice to see work improving the process!
Can today’s LLMs generate rubrics that reliably evaluate tomorrow’s smarter models? Automated rubric generation can save enormous expert effort, but prior work didn’t evaluate responses from LLMs released after the rubric-generator model.
Our new paper investigates how to best scale test-time compute for this, developing a new rubric generation pipeline called Rubricon. Rather than just relying on LLM self-critique, Rubricon grounds rubric-refinement in how candidate rubrics actually score example responses.
Compared to 4 other synthetic rubric generation methods across 5 diverse benchmarks, Rubricon rubrics consistently score model responses more like human experts do [1/n]View quoted post on X
