It might be easier to prove Navier-Stokes than to perform large scale formalization! Even with the very very impressive capabilities of LLMs in theorem proving, auto-formalization is not yet solved. OpenAI/math was only able to formalize ~42% of their results. Cheap autoformalization still needs to be worked on.
🚨 AutoformBench: A Lean Formalization Benchmark for Advanced Mathematics
Our team at @AIatMeta is excited to release AutoformBench, a new formalization benchmark that pushes the Lean 4 coding capabilities of frontier models to their limits.
While SOTA models have become extremely competent at single-shot problem-solving, formalizing mathematics at scale and cost-effectively remains difficult. For instance, only about 42% of the results in openai/math could be formalized before release.
Furthermore, it is not clear and publicly known how much it costs to formalize big results, which is critical information if formalization is to become a part of the peer review process. This is the focus of AutoformBench.
The benchmark comprises 341 proving tasks drawn from diverse graduate- and research-level mathematical sources and formalized by human experts. Each task requires writing a substantial amount of auxiliary code, including lemmas, definitions, and more, placing the emphasis on models’ ability to build formal infrastructure rather than on mathematical cleverness alone.
The result is a demanding test: even the strongest models require tens of millions of tokens to complete 50% of the tasks (roughly $10 to $100 at current API rates).
Paper (arXiv coming soon): github.com/facebookresear…
Code: github.com/facebookresear…
Joint work by myself, Ahmad Rammal (@Ahmad3Rammal) and Vivien Cabannes, together with Niket Patel (@niketnpatel), Fabian Gloeckle (@FabianGloeckle), Bartosz Piotrowski, Amaury Hayat (@Amaury_Hayat), Remi Munos (@MunosRemi) and Julia Kempe (@KempeLab) from @AIatMeta, @NYUDataScience, and Ecole des Ponts. (1/6)View quoted post on X
