SorryDB

SorryDB Leaderboard

SorryDB is a dataset of open proof obligations drawn from real-world Lean formalization projects, designed as a benchmark for measuring the effectiveness of AI theorem provers for day-to-day Lean practitioners.

Snapshot · SorryDB_2601 · January 2026 i A fixed slice of 1,000 tasks from the SorryDB dataset. New snapshots will follow.
Loading leaderboard…

Evaluation pipeline

SorryDB continuously monitors active Lean projects listed on Reservoir. For every open sorry it finds, it records the repository, commit, Lean version, and source location so the task can be reproduced locally.

Each strategy then proposes a proof to replace the sorry, and an independent verifier compiles the candidate inside the original project to check whether the proof closes.

Evaluation Paper

An evaluation of current SOTA models on SorryDB

SorryDB: Can AI Provers Complete Real-World Lean Theorems?

Accepted at ICML 2026

Austin Letson1, Leopoldo Sarra1, Auguste Poiroux3,4, Oliver Dressler, Paul Lezeau5,6, Dhyan Aranha2,7, Frederick Pu10, Aaron Hill, Miguel Corredera Hidalgo8, Julian Berman9, George Tsoukalas11, Lenny Taelman2

1Axiomatic AI · 2University of Amsterdam · 3Math, Inc. · 4EPFL · 5Imperial College · 6The London School of Geometry and Number Theory · 7Côte d'Azur University · 8ENSEIRB-MATMECA, INP-Bordeaux · 9Columbia University · 10University of Toronto · 11The University of Texas at Austin

Abstract We present SorryDB, a dynamically-updating benchmark of open Lean tasks drawn from 78 real world formalization projects on GitHub. Unlike existing static benchmarks, often composed of competition problems, hillclimbing the SorryDB benchmark will yield tools that are aligned to the community needs, more usable by mathematicians, and more capable of understanding complex dependencies. Moreover, by providing a continuously updated stream of tasks, SorryDB mitigates test-set contamination and offers a robust metric for an agent's ability to contribute to novel formal mathematics projects. We evaluate a collection of approaches, including generalist large language models, agentic approaches, and specialized symbolic provers, over a selected snapshot of 1000 tasks from SorryDB. We show that current approaches are complementary: even though an agentic approach based on Gemini Flash is the most performant, it is not strictly better than other off-the-shelf large-language models, specialized provers, or even a curated list of Lean tactics.
Read the paper →