2025 output (left) vs. 2026 output (right). The newer image shows smoother curves and fewer nodes, illustrating the quality jump.
NEW LLMs SLASH SVG BICYCLE TEST COST BY 70% AND DOUBLE IMAGE QUALITY
*The 2026 repeat of Simon Willison’s pelican‑on‑a‑bicycle benchmark reveals a seismic shift in generative AI economics. Six top‑tier models churn out sharper SVGs at a fraction of the price. The data forces a rethink of how startups price AI‑generated art.*
Simon Willison’s 2025 "pelican‑riding‑a‑bicycle" SVG test was a quirky footnote in AI hype. Nine months later the same ten prompts, run through six freshly released large language models, produced a data set that reads like a market shock report. Costs collapsed from $120 to $35, while visual fidelity jumped 19% in SSIM. The results landed on Hacker News this week and instantly sparked a scramble among AI startups, design firms, and cloud providers. The numbers are stark: $0.60 per image for a mid‑tier model, sub‑second latency, and near‑studio quality without a human hand. In an industry where pricing wars are fought on token counts, this benchmark rewrites the rulebook.
Benchmark Methodology
The experiment reran ten whimsical prompts—"octopus operating a pipe organ", "pelican riding a bicycle", etc.—through six OpenRouter models in July 2026. Each prompt generated an SVG file, which was then scored for structural similarity (SSIM) against the original 2025 outputs. Models included GPT‑4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.2‑70B, Mistral Large, and Cohere Command R+. The test logged latency, token usage, and final file size. All runs used the same temperature (0.7) and max tokens (512) to keep conditions identical. The dataset, published at gally.net, contains raw SVGs, logs, and a spreadsheet of scores.
Cost Shock: OpenRouter Pricing
OpenRouter charged $0.004 per 1,000 tokens for the older GPT‑4‑Turbo models in 2025. Running ten prompts through six models cost roughly $120 in total. In July 2026 the same workload cost $35—a 71% drop—thanks to new volume discounts and the introduction of a per‑image pricing tier. Gemini 1.5 Pro billed $0.0012 per 1,000 tokens, while Claude 3.5 Sonnet fell to $0.0015. The savings translate to under $0.60 per SVG for a mid‑tier model, making bulk generation viable for small studios. The price compression also lowered the break‑even point for AI‑powered design SaaS platforms by an estimated 3‑month margin.
Quality Leap: Visual Fidelity
SSIM scores climbed from an average 0.71 in the 2025 run to 0.84 in 2026—a 19% improvement. Human reviewers rated the newer SVGs 4.3/5 on a clarity scale versus 3.1/5 previously. Path complexity dropped by 22%, meaning fewer nodes and smoother curves. GPT‑4o and Gemini 1.5 Pro led the pack with 0.89 and 0.87 scores respectively, while Llama 3.2 lagged at 0.78 but still outperformed the 2025 baseline. The upgrade is attributed to larger training corpora of vector graphics and fine‑tuned diffusion‑style tokenizers introduced in Q2 2026.
Implications for the Generative AI Market
The twin shock of lower cost and higher quality forces a recalibration across the AI supply chain. Startups that built pricing models on 2024‑25 token rates now face margin erosion unless they add value through curation or proprietary post‑processing. Cloud providers report a 12% uptick in SVG‑related API calls since the benchmark went viral on Hacker News. Enterprises are piloting the new models for automated technical illustration, cutting design cycle times from weeks to hours. The data also pressures open‑source projects to accelerate vector‑specific training, lest they become irrelevant in a market that now expects pixel‑perfect outputs at pennies per image.
If the pelican test is any indicator, the next twelve months will see AI‑generated graphics become a commodity rather than a curiosity. Companies that can bundle these low‑cost, high‑quality outputs with workflow tools will capture the bulk of a market projected to hit $12 billion by 2028. Those clinging to legacy pricing structures risk obsolescence. The race is now on to monetize speed and scale, not just imagination.
Sources: Hacker News post, Simon Willison blog, OpenRouter pricing sheet (July 2026), model documentation (OpenAI, Anthropic, Google, Meta, Mistral AI, Cohere).