Figure 1 from the FrontierHarness Eval v1.1 report plots each harness’s cost per task against its pass rate, highlighting the Pareto‑optimal line.
*RUNTA and Baseten built the largest‑scale harness leaderboard to date, testing 22 harnesses on five models across 62 tasks. The data shows that the optimal harness shifts dramatically with each model, reshaping cost‑performance calculations for AI agents.*
RUNTA partnered with Baseten to launch FrontierHarness Eval v1.1, a benchmark that dwarfs its predecessor. The new suite runs 83 billion tokens through 27,280 agent executions, covering 22 harness configurations, five language models, and 62 terminal tasks. The goal was two‑fold: determine whether a model change warrants a harness change, and identify the best‑value harness for each model. The experiment expands the earlier single‑model leaderboard by 76 times, delivering a data set large enough to expose cost and pass‑rate dynamics that were previously invisible. The findings overturn the assumption that a single harness can dominate across model families.
The v1.1 run evaluated 22 harness‑model pairs across five models: GLM‑5.3, GLM‑Flash, DeepSeek‑V4.1‑Flash‑1017, Muse, and Kimi‑K3. Each pair tackled 62 terminal tasks with four attempts, totaling 27,280 runs and 83 billion tokens. Harnesses kept default settings and maximum reasoning effort to mirror everyday use. Cost metrics included average cost per task and cost per pass, the latter derived by dividing average cost by pass rate. The orange Pareto line in Figure 1 marks harnesses that are unbeaten in both dimensions. Over all models, four harnesses sit on that line: OpenCode v2 (lowest cost per task at $0.561), Pi, Codex CLI, and ZCode (highest pass rate at 61.9%).
The data says yes. Each model’s top‑value harness differed, and cost rankings shifted dramatically. For GLM‑5.3, Codex CLI delivered the best cost‑per‑pass at $0.803, but the same harness fell to 12th place with Muse and 10th with Kimi‑K3. Median harness rank moved 11 of 22 slots when the model changed. Cost per task varied by a factor of 1.8 to 5.0 across harnesses for a fixed model, far exceeding the 1.2‑1.8 range expected by chance. These swings prove that a cheap harness on one model can become expensive on another, making model‑specific harness selection essential for budget‑conscious deployments.
RUNTA identified a single best‑value harness for each model using a cost‑per‑pass rule, allowing a 30% cost premium if the pass rate improves by at least five points. GLM‑5.3 pairs best with Codex CLI. GLM‑Flash’s top harness is Pi. DeepSeek‑V4.1‑Flash‑1017 aligns with ZCode. Muse performs best with Oh My Pi, while Kimi‑K3 finds its optimum in Alma. The rule swapped ZCode for OpenCode v1 on DeepSeek (25% higher cost, 7.7‑point pass‑rate gain) and Alma for Oh My Pi on Kimi‑K3 (9% higher cost, 6.5‑point gain). Across all models, Codex CLI achieved the lowest overall cost per pass at $0.937, while Antigravity CLI was the most expensive at $1.63.
To gauge stability, the 62 tasks were resampled 4,000 times. Codex CLI retained the GLM‑5.3 lead in 99.6% of draws. The other winners showed lower certainty: Oh My Pi with Muse (53%), ZCode with DeepSeek (47%), Pi with GLM‑Flash (42%), and Alma with Kimi‑K3 (25%). A two‑way ANOVA attributed 92.8% of cost variance to the model, 2.3% to the harness, and 4.9% to the model‑harness pair. With the model fixed, 68% of the remaining variation stems from the pair or noise, confirming that model choice dominates cost while harness selection still moves the needle by up to fivefold.
OpenCode v2 records the lowest average cost per task at $0.561, placing it on the Pareto‑optimal line for all models.
Codex CLI paired with GLM‑5.3 delivers the best cost‑per‑pass at $0.803 and ranks first for that model.
Across all models, Codex CLI achieves the lowest overall cost per pass at $0.937.
FrontierHarness Eval v1.1 shatters the myth of a one‑size‑fits‑all harness. By exposing how model selection reshapes both cost and success rates, the benchmark forces developers to treat harness choice as a strategic lever, not a static setting. As AI agents proliferate in high‑stakes environments, the ability to pair the right harness with the right model could become a competitive advantage, driving tighter budgets and higher reliability across the industry.
Sources: Hacker News (https://runta.com/blog/introducing-frontierharness-eval-v1-1/)