The original C code for range length calculation (left) and its Lean counterpart (right) produced by the AI formalization pipeline.
AI AGENTS FORMALIZE CPYTHON, UNVEIL 418 BUGS
*AI-driven Lean rewrites of CPython's C code have produced 613,318 lines of machine‑checked proofs. The effort exposed 418 bugs, 55 of which can crash the interpreter.*
A swarm of AI agents has taken on CPython, the reference implementation that powers almost every Python program on the planet. By translating each C routine into Lean, a language built for machine‑checked proofs, the team forced the interpreter’s hidden logic into a precise, testable form. The experiment, run on CPython 3.15.2, produced over 600 k lines of formal code and surfaced 418 bugs, including dozens that can crash the runtime. The findings arrive at a moment when the Python community wrestles with reliability in a language that has never had a formal specification.
How did the AI formalization process work?
The team pointed Opus 5.5 at CPython 3.15.2. Each function underwent four identical steps. First, an AI copied the C source line‑by‑line into Lean, preserving error messages and edge‑case arithmetic. Second, the Lean copy and the real interpreter ran side‑by‑side on at least 50,000 tricky inputs; any mismatch flagged the copy as wrong. Third, the agents wrote a proof stating what must always hold and let Lean verify it. Fourth, a second, independent AI reviewed the copy, the test results, and the proof, trying to break it. Only functions that survived all four stages counted. So far, 4,071 functions have passed this pipeline.
What scale has been reached and what parts of CPython are covered?
The project has generated 613,318 Lean lines across 1,539 files, embedding 18,898 theorems and lemmas. Coverage varies by subsystem: 16% of the interpreter core, 10% of built‑in types, 35% of C modules, 19% of the parser, 31% of grammar rules, and 9% of the standard library have been copied, tested, proved and accepted. Code that deals only with memory bookkeeping was excluded. The effort shows that formal verification can scale into a mature, multi‑million‑line codebase.
Which bugs were uncovered and how serious are they?
During proof validation, the system logged 418 bug reports. Most were minor, but roughly 55 bugs could crash the interpreter or corrupt memory from ordinary Python code. The bugs emerged from obscure corner cases that the reference manual does not cover, such as objects mutating dictionaries during a search or errors thrown mid‑update. The formal proofs forced developers to articulate exact behavior in these edge cases, exposing the hidden defects.
What limitations remain and what comes next?
Early attempts produced weak proofs that let broken functions slip through—for example, a proof that merely asserted a non‑negative return value. Reviewers now require a sanity question for each proof: could a broken version still satisfy it? This stricter criterion has raised acceptance rates dramatically, as shown by the review charts. However, functions that lack a realistic copy or that involve low‑level memory tricks remain outside the scope. Future work will extend coverage, tighten proof requirements, and integrate the verified components back into the main CPython release.
Frequently asked questions
How many Lean lines were generated in the CPython formalization?
The project produced 613,318 lines of Lean across 1,539 files.
How many bugs did the formalization uncover?
The proofs turned up 418 bug reports, about 55 of which can crash the interpreter or corrupt memory.
What percentage of CPython's interpreter core has been formally verified?
Sixteen percent of the interpreter core has been copied, tested, proved and accepted.
The CPython formalization project proves that AI can automate the painstaking work of proof generation, but it also shows that human oversight remains essential. As the coverage expands, the Python ecosystem may finally gain a mathematically sound definition of its behavior, reducing the surprise bugs that have long haunted developers. The next milestone will be integrating the verified modules back into the official interpreter, turning proof into practice.
Sources: Hacker News article "Formalizing CPython" (https://broyojo.com/python-formalization/)