← Back to BLACKWIRE EMBER BUREAU AI COST EFFICIENCY Graph showing cost per task versus pass rate for multiple AI harnesses across different models

Figure 1 from the FrontierHarness Eval v1.1 report plots each harness’s cost per task against its pass rate, highlighting the Pareto‑optimal line.

FRONTIERHARNESSEVAL V1.1 REVEALS MODEL‑SPECIFIC BEST HARNESS

*RUNTA and Baseten built the largest‑scale harness leaderboard to date, testing 22 harnesses on five models across 62 tasks. The data shows that the optimal harness shifts dramatically with each model, reshaping cost‑performance calculations for AI agents.*

By EMBER Bureau - BLACKWIRE  |  November 17, 2026, 19:00 CET  |  FrontierHarness, AI harness benchmark, model cost efficiency, pass rate, AI agents

RUNTA partnered with Baseten to launch FrontierHarness Eval v1.1, a benchmark that dwarfs its predecessor. The new suite runs 83 billion tokens through 27,280 agent executions, covering 22 harness configurations, five language models, and 62 terminal tasks. The goal was two‑fold: determine whether a model change warrants a harness change, and identify the best‑value harness for each model. The experiment expands the earlier single‑model leaderboard by 76 times, delivering a data set large enough to expose cost and pass‑rate dynamics that were previously invisible. The findings overturn the assumption that a single harness can dominate across model families.

Scale and scope of the v1.1 benchmark

The v1.1 run evaluated 22 harness‑model pairs across five models: GLM‑5.3, GLM‑Flash, DeepSeek‑V4.1‑Flash‑1017, Muse, and Kimi‑K3. Each pair tackled 62 terminal tasks with four attempts, totaling 27,280 runs and 83 billion tokens. Harnesses kept default settings and maximum reasoning effort to mirror everyday use. Cost metrics included average cost per task and cost per pass, the latter derived by dividing average cost by pass rate. The orange Pareto line in Figure 1 marks harnesses that are unbeaten in both dimensions. Over all models, four harnesses sit on that line: OpenCode v2 (lowest cost per task at $0.561), Pi, Codex CLI, and ZCode (highest pass rate at 61.9%).

Should you switch harnesses when you change models?

The data says yes. Each model’s top‑value harness differed, and cost rankings shifted dramatically. For GLM‑5.3, Codex CLI delivered the best cost‑per‑pass at $0.803, but the same harness fell to 12th place with Muse and 10th with Kimi‑K3. Median harness rank moved 11 of 22 slots when the model changed. Cost per task varied by a factor of 1.8 to 5.0 across harnesses for a fixed model, far exceeding the 1.2‑1.8 range expected by chance. These swings prove that a cheap harness on one model can become expensive on another, making model‑specific harness selection essential for budget‑conscious deployments.

"You should consider switching harnesses when switching models to maximize your experience."

Best‑value harnesses per model

RUNTA identified a single best‑value harness for each model using a cost‑per‑pass rule, allowing a 30% cost premium if the pass rate improves by at least five points. GLM‑5.3 pairs best with Codex CLI. GLM‑Flash’s top harness is Pi. DeepSeek‑V4.1‑Flash‑1017 aligns with ZCode. Muse performs best with Oh My Pi, while Kimi‑K3 finds its optimum in Alma. The rule swapped ZCode for OpenCode v1 on DeepSeek (25% higher cost, 7.7‑point pass‑rate gain) and Alma for Oh My Pi on Kimi‑K3 (9% higher cost, 6.5‑point gain). Across all models, Codex CLI achieved the lowest overall cost per pass at $0.937, while Antigravity CLI was the most expensive at $1.63.

Statistical confidence and cost drivers

To gauge stability, the 62 tasks were resampled 4,000 times. Codex CLI retained the GLM‑5.3 lead in 99.6% of draws. The other winners showed lower certainty: Oh My Pi with Muse (53%), ZCode with DeepSeek (47%), Pi with GLM‑Flash (42%), and Alma with Kimi‑K3 (25%). A two‑way ANOVA attributed 92.8% of cost variance to the model, 2.3% to the harness, and 4.9% to the model‑harness pair. With the model fixed, 68% of the remaining variation stems from the pair or noise, confirming that model choice dominates cost while harness selection still moves the needle by up to fivefold.

Frequently asked questions

Which harness has the lowest cost per task across all models?

OpenCode v2 records the lowest average cost per task at $0.561, placing it on the Pareto‑optimal line for all models.

What is the best harness for the GLM‑5.3 model?

Codex CLI paired with GLM‑5.3 delivers the best cost‑per‑pass at $0.803 and ranks first for that model.

How much does the overall cheapest harness cost per pass?

Across all models, Codex CLI achieves the lowest overall cost per pass at $0.937.

FrontierHarness Eval v1.1 shatters the myth of a one‑size‑fits‑all harness. By exposing how model selection reshapes both cost and success rates, the benchmark forces developers to treat harness choice as a strategic lever, not a static setting. As AI agents proliferate in high‑stakes environments, the ability to pair the right harness with the right model could become a competitive advantage, driving tighter budgets and higher reliability across the industry.

Sources: Hacker News (https://runta.com/blog/introducing-frontierharness-eval-v1-1/)