AI Coding Without the Waste

There's no single "best" model. There's the right model for your budget and quality bar.

What to avoid

• Assuming price predicts quality. Opus is ~1.7× Sonnet's per-token rate at PaleBlueDot gateway rates ($25/M vs $15/M output, $5/M vs $3/M input) — modest on paper. Opus scored lower in V1 (8.22 vs 8.62) but topped V2's builder ranking (7.77). Across V1 + V2 builds, Opus drew $145 of the $377 builder API spend — the highest of any single builder. Per-call cost (driven by call length + context depth) is a much bigger factor than per-token rate. Model rankings also shift with evaluation methodology; always test in your own setup.
• o3-mini for creative coding. Scored 5.23 — dead last by 1.55 points. Honest caveat: o3-mini was released in January 2025, making it over 15 months old at benchmark time, and it's a lightweight reasoning model by design (the smallest of the o-series). Its poor showing is as much about age and weight class as architecture — a genuinely useful snapshot of how fast the frontier moves. With hindsight we should have tested o5-mini instead (I thought we had, but the pipeline picked up o3-mini); if you want a reasoning-model comparison for coding, start there.
• Extrapolating from models we barely tested. GPT-5.4 Nano appeared only in two V1 runs and was not included in V2. We don't have enough evidence to recommend or dismiss it — just don't treat the limited data as strong evidence.
• Assuming Gemini Pro is bad because V1 said so. It isn't. V1's 6.78 consensus was depressed by the Flash judge inflating other models more than Pro. Clean V1 scores put Pro at 7.15, same as Flash. V2 (cleaner methodology) put Pro at 7.45 — #2.

Where to start — which subscriptions to buy

If you're new to AI coding and want to set yourself up properly without overpaying, the data points in a clear direction.

Practical pipeline pairings (planner × builder)

If you're building agent systems where one model writes a spec and another model executes it, pair carefully. The wrong combination can make a strong builder produce broken output.

Pairings to avoid

• Opus → Haiku, Mini, or Nano. Small builders choke on long specs. V2 Haiku Snake scored 7.93 with a light spec, 1.43 with Opus's 12 KB spec. Same model, same game. Spec complexity must be calibrated to builder capability.
• Opus → Sonnet for well-known patterns. Sonnet loses 0.28 points with detailed specs. Use Gemini Pro as the planner instead — lighter spec, similar quality output.
• GPT-5.4 builder with no detailed spec. GPT-5.4 builders gain +0.74 from a detailed plan — they're the only model that strongly needs one. Without a structured spec, GPT-5.4 underperforms its tier.
• Haiku or GLM-5 as planners with any premium builder. Neither produces specs that improve premium builder output, and you're paying twice.

We ran a direct experiment: five planners wrote specifications for the same 10 games, ranging from 29,000-byte engineering blueprints (Opus) to 90 bytes (just the game name). Eight builder models built from each spec.

The result, confirmed by 3-judge static QA: Builders given only the game name averaged 6.86/10 — higher than the 6.70 average from detailed specifications. Most builders scored worse with specs; GPT-5.4 Mini lost over a full point.

This doesn't mean specs are always useless. It means the model's training data already includes the "spec" for well-known tasks. When you write a detailed specification for Pac-Man, you're largely restating what the model already knows — and sometimes constraining it in ways that make the output worse.

When specs DO help

• Novel or domain-specific tasks — anything the model hasn't seen a thousand times
• Non-obvious requirements — specific colour schemes, accessibility constraints, business rules
• Integration points — how this code connects to your existing system
• Constraints the model wouldn't assume — no external dependencies, must run offline, maximum 50KB

When specs hurt

• Over-specifying implementation details for well-known patterns — the model's default approach may be better than yours
• Contradictory requirements — long specs are more likely to contain contradictions that confuse the model
• Template-driven specs — forcing a model to fill in 12 sections when 3 would do

"Vibe coding" — coined by Andrej Karpathy — means giving an AI a loose description and letting it build whatever it interprets. No detailed spec, no iteration, no code review. Just vibes.

Our benchmark is essentially a 580-build vibe coding experiment. Every game was built in a single pass with no human intervention. The results:

• Simple games (Snake, Pong) — vibe coding works brilliantly. Average scores above 8.5. Most builds are playable and recognisable.
• Medium games (Tetris, Breakout) — usually playable but with notable issues: missing features, visual shortcuts, occasional bugs. Scores 7.5–8.5.
• Complex games (Pac-Man, Donkey Kong) — hit or miss. Some impressive implementations, some complete failures. Scores 6.0–7.0 with high variance.

The vibe coding spectrum

The right level depends on what you're building and how much you can afford to get wrong. Vibe coding a prototype? Go for it. Vibe coding your payment system? Please don't.

AI coding costs are deceptive. The per-token prices look small until you multiply by volume, iterations, and the invisible cost of debugging bad output.

Trap 1 — Premium by default

Most people start with the "best" model and never test cheaper alternatives. In our benchmark, Gemini Flash scored ~90% of Sonnet's AI-judge consensus (7.74 vs 8.62 in V1) at PaleBlueDot rates that are ~5× cheaper on output ($3 vs $15 / M tokens) and ~6× cheaper on input ($0.50 vs $3 / M). For volume generation where small quality drops are acceptable, the math favours Flash heavily.

The honest caveat: AI judges read the code and score whether it runs, looks right, and handles controls. They don't meaningfully assess readability, maintainability, or code taste. Human evaluators often find a wider gap than the 10% suggests — Flash's output is more likely to be oddly structured, sparsely commented, or built on shortcuts that work but aren't good code. If you're generating once and discarding, Flash is obvious value. If the output will be read, extended, or live in production, the premium tier often buys more than the score says.

Trap 2 — Paying for reasoning you don't need

Reasoning models think step-by-step before generating. That's brilliant for maths and logic puzzles, expensive for creative work that wants whole-system generation. In our benchmark, the only reasoning model in the panel — o3-mini — scored dead last (5.23/10).

The honest context: o3-mini was released January 2025, making it over 15 months old at benchmark time, and it's a lightweight reasoning model by design. Its last-place finish is as much about age and weight class as it is about "reasoning vs creative coding" — you can read this as a snapshot of how fast the frontier has moved in 15 months. In hindsight we should have used o5-mini (I genuinely thought we had; turns out the pipeline picked up o3-mini and the mistake only surfaced in analysis). If you're comparing a reasoning model for code generation today, start with o5-mini or similar — not o3-mini.

The narrow takeaway: don't pay premium for o3-mini when a modern budget model will outperform it. The broader question — whether reasoning architectures genuinely hurt creative coding — needs more than one old data point to answer.

Trap 3 — Iteration without evaluation

The most expensive habit: re-prompting a model repeatedly without checking if the output is actually getting better. Each iteration costs tokens. If you're not measuring quality between iterations, you're paying for random walks.

When we asked AI judges to evaluate AI-built games, they disagreed significantly. V1's Gemini Flash judge rated nearly everything 9–10, inflating scores by ~1.8 points. V2 replaced it with Gemini Pro — and the entire leaderboard shifted as a result. The judge panel you choose changes your scores even if the underlying quality is identical.

The lesson generalises beyond our benchmark: AI models are unreliable evaluators of their own output. They're biased toward marking things as "correct" and they can't detect the kinds of bugs that only show up when you actually use the software.

What automated testing catches

• Syntax errors and crashes
• Missing elements (no canvas, no game loop)
• Static analysis issues
• Basic functionality checks (does the score change? does the game start?)

What only a human catches

• The ghost AI in Pac-Man doesn't actually chase you
• Tetris pieces rotate but clip through the floor
• The game is technically playable but not fun
• The scoring works but the difficulty never increases
• It looks like Pac-Man drawn by a five-year-old

Zero-intervention AI coding (our benchmark methodology) produced average scores of 7.5/10 across 580 builds. That's "functional but flawed" territory. The question isn't whether human intervention helps — it obviously does — but when the return on your time is highest.

High-value interventions

• Reviewing the architecture before coding starts. If the model's planned approach is wrong, fixing it early saves everything downstream.
• Testing the output. 5 minutes of manual testing catches bugs that 3 AI judges missed.
• Specifying non-obvious constraints. "Must work offline", "max 50KB", "needs to support screen readers" — things the model won't assume.

Low-value interventions

• Micro-managing implementation. If the model knows how to build Pong, telling it which collision algorithm to use probably makes things worse.
• Writing exhaustive specs for well-known patterns. Our data shows 29K-byte specs performed no better than 3-line prompts for known games.
• Iterating on the prompt instead of the code. If the output is 80% right, it's faster to fix the code than to craft the perfect prompt that generates perfect code.

AI coding tools fall into a few categories:

This benchmark used OpenClaw as the agent framework — it handles session management, model routing, and artefact verification. The games themselves were built entirely by the AI models; OpenClaw just orchestrated the process.

If this guide was useful, here's how to go deeper:

• Read the full research paper — all 580 builds analysed in detail with interactive charts and playable game embeds. Read the paper →
• Play the games — every build is a single HTML file you can open in a browser. See what AI-generated code looks and feels like. Open the library →
• Try it yourself — pick a simple project (a to-do app, a calculator, a game) and build it with 3 different models. Compare the output. This teaches you more than any guide.
• Track your costs — before you start, decide your budget. Log tokens and costs from the first build. You'll be surprised where the money goes.
• Test everything — the single most impactful habit. Run the code. Click the buttons. Check the edge cases. AI output looks right more often than it is right.