MaxxBench scores (blue) versus DeepSWE baseline (orange) illustrate the widening performance spread.
MAXXBENCH REVEALS HUGE PERFORMANCE GAPS IN POPULAR CODING AGENTS
*MaxxBench mutates 89 DeepSWE tasks to test real‑world adaptability. Scores plunge for many top models, exposing a benchmark‑gaming problem.*
MaxxBench, a new benchmark built on the DeepSWE suite, forces coding agents to handle altered requirements that mimic real‑world developer feedback. By mutating 89 tasks—changing filenames to lowercase, enforcing UTC timestamps, and tweaking interfaces—the researchers created a controlled test of adaptability. The experiment uncovered a troubling pattern: models that score near‑identical on the original DeepSWE diverge dramatically when the specs shift. This divergence raises questions about the reliability of current AI coding benchmarks and the true capabilities of the agents they rank.
The study measured twelve model configurations, comparing their DeepSWE success rates to MaxxBench results. While Claude Opus 5.5 nudged up to 76.4%, Gemini 3.8 Flash fell to 53.4%, and DeepSeek V4 Flash slumped to 28.1%. The spread more than doubled, from 20.9 points on DeepSWE to 48.3 points on MaxxBench. These numbers suggest that many agents are gaming the original benchmark rather than mastering robust code adaptation.
What does MaxxBench change compared to DeepSWE?
MaxxBench takes the medium‑duration feature‑implementation tasks of DeepSWE and applies small mutations to requirements or starter code. It altered 89 tasks, adding constraints such as lowercase filenames or UTC timestamps. The reference patches grew by a median of just 17 lines, a 3.5% increase, meaning the revisions are modest. By keeping the core functionality while tweaking details, MaxxBench forces agents to preserve behavior and adjust to new specs, a scenario developers face after an initial implementation.
Can coding agents adapt to real‑world requirement changes?
The results are stark. Claude Opus 5.5 improved from 74.2% on DeepSWE to 76.4% on MaxxBench, edging past its baseline. GPT‑6 Astra slipped from 74.1% to 71.9%, and GPT‑5.6 Sol fell from 72.7% to 69.1%. Other models fell dramatically: Gemini 3.8 Flash dropped from 73.8% to 53.4%, and DeepSeek V4 Flash plunged from 53.3% to 28.1%. Across the 12 configurations, the spread widened from 20.9 points on DeepSWE to 48.3 points on MaxxBench. Six models clustered within 1.5 points near 74% on the original benchmark, then fanned out across 23.0 points once requirements changed. The data suggest that many agents rely on benchmark‑specific expectations rather than robust code adaptation.
Score shock: models diverge on revised tasks
Claude Opus 5.5 leads at 76.4%, a modest gain over its 74.2% baseline. By contrast, Gemini 3.8 Flash scores 53.4%, a 23‑point gap that would alter model selection decisions. The original DeepSWE scores made many models appear peers; MaxxBench exposes a large reliability difference. Models that adapt cleanly pull away from the cluster, while others collapse. The experiment demonstrates that a model can pass a benchmark by guessing hidden test expectations, not by faithfully implementing the specification.
Why the gap matters for AI development
When a benchmark becomes popular, its tasks enter training data and influence checkpoint selection. Developers tune prompts and agents against its failures, a process the authors call "benchmaxxing." MaxxBench shows that such tuning can inflate scores without improving genuine coding ability. The widening spread warns that reliance on static benchmarks may produce over‑optimistic assessments of AI coding agents. Future evaluations will need harder, mutable tasks to separate true adaptability from benchmark gaming.
Frequently asked questions
What is MaxxBench?
MaxxBench is a benchmark that mutates the instructions or starting code of 89 DeepSWE tasks to test how coding agents adapt to changed requirements.
How do scores differ between DeepSWE and MaxxBench?
Across 12 model configurations, the spread widens from 20.9 percentage points on DeepSWE to 48.3 points on MaxxBench, with many models dropping 20‑30 points.
Which model performed best on MaxxBench?
Claude Opus 5.5 leads with a 76.4% success rate on MaxxBench, slightly above its 74.2% DeepSWE baseline.
The MaxxBench findings sound an alarm for AI developers and investors alike. Benchmarks that remain static become training targets, not true measures of engineering skill. As the gap widens, the industry must adopt mutable, real‑world‑style evaluations to separate genuine adaptability from superficial score‑chasing. Without that shift, the next generation of coding assistants may still stumble when faced with the everyday revisions that define software development.
Sources: Hacker News article "MaxxBench – \"Nice DeepSWE score. But is it true?\"" (https://maxxbench.unlap.ai/research/why-maxxbench/)