Claude Sonnet 5.5 tops the DAYJOB: Tax leaderboard with a 10.8% pass rate, highlighting AI's struggle with professional tax work.
AI TAX AGENTS FAIL TO CATCH DEADLINE IN CONDO CASE, PASS RATES BELOW 11%
*DAYJOB: Tax pushes AI agents into real‑world tax work. Fifty messy assignments expose gaps in professional judgment. Top models clear fewer than one in ten tasks.*
Surge AI’s DAYJOB: Tax benchmark forces artificial agents to act like human tax professionals, from the first client request to the final deliverable. The test mirrors the chaos of real filings: incomplete instructions, conflicting records, and hidden deadlines that can swing millions of dollars. By embedding 50 distinct environments and 1,789 files, the benchmark measures not just calculations but the initiative to hunt down consequential issues.
The benchmark’s first public case involved Anna, who planned to sell her condo in October 2026 or January 2027. An AI agent correctly confirmed that either date met the residence requirement for a $250,000 exclusion, yet it omitted the two‑year residency deadline that expires in August 2027. Missing that deadline could strip Anna of the tax break entirely, exposing a fatal gap in the agent’s professional judgment.
What does the DAYJOB: Tax benchmark test?
The benchmark assembles 50 tax environments across eight practice areas, from individual returns to multistate corporate disputes. Agents receive client requests, PDFs, Excel workbooks, Word files, emails, images, CSVs, JSON and raw text—totaling 1,789 workspace files with a median of 30 per assignment. Tasks range from filing complete returns to investigating undocumented transactions, such as a handwritten receipt that must be reconciled with inventory costs. Each assignment undergoes a three‑layer expert review, and success requires meeting a 95% rubric that checks calculations, evidence, consequential issues and final deliverables.
Can AI agents reliably spot hidden tax deadlines?
Anna’s condo scenario proved decisive. Across five attempts, the Astra model never flagged the August 2027 residency cutoff, while Opus identified it every time. The benchmark treats such missed deadlines as a failure to address consequential issues, a core component of the pass criteria. Overall, only 10.8% of attempts by the leading model met the 95% threshold, underscoring that current AI struggles to surface critical, client‑unaware risks that seasoned tax advisers routinely catch.
How did leading models perform?
Claude Sonnet 5.5 topped the leaderboard with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. Pass@1 is measured against a 95% rubric, meaning most attempts fell short on calculations, evidence handling, or identifying consequential issues. When plotted against average cost per trial, Sonnet 5.5 and Opus 5.5 sit on the high‑performance end of the Pareto frontier, indicating that higher expense does not guarantee substantially better outcomes.
Why the low pass rates matter for firms?
A sub‑11% success rate signals that firms cannot rely on AI alone for tax compliance or advisory work. Missed deadlines, mis‑read receipts, or overlooked multistate obligations can trigger audits, penalties, and lost deductions. The benchmark’s diverse file types—PDFs, Excel sheets, handwritten images—mirror the unstructured data tax departments face daily. Until models consistently demonstrate professional judgment, human oversight remains essential, and firms must treat AI output as a draft, not a final filing.
Frequently asked questions
What is the pass rate of the top AI model on DAYJOB: Tax?
Claude Sonnet 5.5 leads with a 10.8% pass rate.
How many environments and files are in the DAYJOB: Tax benchmark?
The benchmark contains 50 environments and 1,789 workspace files.
Which AI model identified the condo residency deadline in all attempts?
Opus identified the deadline in all five attempts, while Astra never did.
DAYJOB: Tax lays bare the gap between raw computational power and the nuanced decision‑making tax professionals provide. As AI models inch up the Pareto frontier, the industry must decide whether incremental cost gains justify deployment in high‑stakes filing environments. For now, the benchmark warns that without robust judgment, AI remains a risky substitute for seasoned expertise.
Sources: [Surge AI blog Dayjob: Tax — Benchmarking AI Agents on Professional Tax Work]