Yesterday, we released the Conceptual Reasoning Index (CRI). The benchmark that we give most weight in the CRI is LMCA (Language Model Conceptual Argumentation). Our team of conceptual researchers poured 1000+ hours into generating its data.
We take position texts, arguments against these position texts, and then painstakingly hand rate the quality of the arguments according to a detailed rubric. We use those ratings to measure how good models are at rating arguments. We focus on argument evaluation because even on questions where we have little access to ground truth, such as on AI risk and philosophy, there is often much more agreement on which arguments are good.
We compare human ratings against each other for validation and to estimate ceiling performance. We find that human inter-rater agreement is much higher than agreement between models and humans. You can also read more about our validation process on our website (conceptualreasoning.ai/lmca). You can see in the graph that LMCA scores also correlate nicely with other capability scores.
You can also use the LMCA dataset to evaluate how well models can *write* arguments against position texts in our dataset, via an LLM grader prompted with human ratings of other arguments against the same position text. We will write more about this in the future.
Apply for dataset access here: forms.gle/iu9DpgyZd46qUC…
Mid-development arXiv paper (working on an update): arxiv.org/abs/2607.27499
We take position texts, arguments against these position texts, and then painstakingly hand rate the quality of the arguments according to a detailed rubric. We use those ratings to measure how good models are at rating arguments. We focus on argument evaluation because even on questions where we have little access to ground truth, such as on AI risk and philosophy, there is often much more agreement on which arguments are good.
We compare human ratings against each other for validation and to estimate ceiling performance. We find that human inter-rater agreement is much higher than agreement between models and humans. You can also read more about our validation process on our website (conceptualreasoning.ai/lmca). You can see in the graph that LMCA scores also correlate nicely with other capability scores.
You can also use the LMCA dataset to evaluate how well models can *write* arguments against position texts in our dataset, via an LLM grader prompted with human ratings of other arguments against the same position text. We will write more about this in the future.
Apply for dataset access here: forms.gle/iu9DpgyZd46qUC…
Mid-development arXiv paper (working on an update): arxiv.org/abs/2607.27499
We want AIs to be able to help with work to reduce AI risk. But while models do great in domains where reliable feedback is relatively cheap and abundant, like Math and coding, a lot of work on AI risk isn't like that. Instead, we have to rely on good argumentation to answer questions like "does this experiment tell us anything about future models that are much smarter than humans?"
Unfortunately, this kind of work seems much harder to measure (and hence automate). Our team at @redwood_ai developed the Conceptual Reasoning Index (CRI) in collaboration with @AnthropicAI to fix this.
Every single data point in the CRI has been manually checked by a researcher on our team to ensure quality.
This chart shows the performance of each tested company's highest-scoring model plus Fable 5, Muse Spark 1.2, and Gemini Flash 3.6 which are often their company's frontrunners on other capability benchmarks. A score of 0 corresponds to randomising guessing on all three benchmarks and a score of 100 is the highest possible score on all. We estimate 91 to be the true performance ceiling. More info below.
Official leaderboard website which we'll keep up-to-date: conceptualreasoning.aiView quoted post on X