Lighthouse scores vary across frameworks, highlighting performance trade‑offs alongside AI cost.
AI COST OF JAVASCRIPT FRAMEWORKS REVEALED IN BENCHMARK
*A new benchmark measures AI token spend, development time, and Lighthouse scores across ten JavaScript frameworks. Results show a $2.67 low for Hono+htmx and a $4.97 high for Angular, exposing wide cost variance.*
A developer‑focused benchmark has quantified how much artificial‑intelligence assistance costs when building a modest web app with popular JavaScript frameworks. The test used a small Pokémon catalog built via the PokéAPI, a spec‑driven conformance suite, and Opus 5.5 agents instructed to use the latest version of each framework. Each run recorded token spend, lines of code, development time, and Lighthouse performance and accessibility scores. The experiment ran multiple iterations per framework to smooth out the stochastic nature of large language models and the known variance of Lighthouse metrics. The resulting data expose a three‑fold spread in AI‑driven development cost, challenging the community’s vague claims of “blazing fast” or “supercharged” tooling.
How the benchmark was constructed
The author designed a non‑trivial Pokémon app that lists species, types, and moves, then asked Opus 5.5 agents to build it with a specified framework. No CSS frameworks such as Tailwind or Bootstrap were allowed; visual consistency was enforced via spec tokens. Agents received only the functional spec and a high‑effort prompt, without knowledge of upcoming Lighthouse testing. Each framework was run five times, except Angular, which succeeded only three times after two failures. Results were aggregated by taking the median of each set, acknowledging that both LLM output and Lighthouse scores are inherently stochastic.
Which framework costs the least to build with AI?
The median AI cost per run ranged from $2.67 for Hono+htmx (range $2.23–$3.19) up to $4.97 for Angular (range $3.19–$6.82). Astro posted $2.76, Nuxt $2.82, and Vanilla $2.98. Next.js cost $3.14, Qwik $3.27, and TanStack Start $3.54. SolidStart reached $3.79, React Router $3.81, SvelteKit $4.28, and Gracile+Lit $4.65. The spread reflects both framework tooling efficiency and the randomness of token consumption during code generation.
What does the performance data say?
Lighthouse scores varied across runs, with metrics like Largest Contentful Paint shifting between images on successive executions. The author notes that Lighthouse itself can give inconsistent results, making single‑run conclusions unreliable. By averaging multiple runs, the benchmark mitigates this noise but still reports a noticeable variance in performance and accessibility scores among frameworks. The data suggest that lower AI cost does not automatically translate to higher Lighthouse performance.
Why the results matter for developers and investors?
Developers can no longer rely on vague marketing slogans; the benchmark provides concrete cost figures that directly affect project budgets, especially as AI‑assisted coding scales. Investors in web‑technology startups gain a clearer view of operational expense ceilings when AI agents replace human labor. The stochastic nature of both LLM output and Lighthouse metrics also warns against over‑optimistic projections based on single‑run demos. Stakeholders now have a data‑driven baseline to compare framework choices against real AI spend.
Frequently asked questions
What was the median AI cost for building the Pokémon app with Hono+htmx?
The median cost was $2.67 per run, with a range of $2.23 to $3.19.
How many runs were used to calculate each framework’s cost?
Five runs were performed for each framework; Angular had only three successful runs after two failures.
Did the benchmark control for CSS frameworks?
Yes, the test prohibited Tailwind, Bootstrap, and other component frameworks, focusing solely on JavaScript framework performance.
The benchmark strips away hype and delivers hard numbers on AI‑driven development cost across the JavaScript ecosystem. As agents become the default coders, frameworks that minimize token spend while maintaining acceptable performance will dominate budgets. The industry must now treat these metrics as a new KPI, or risk basing decisions on empty vibes.
Sources: [Hacker News – The AI cost of JavaScript frameworks]