Before this:The provider & model landscape
Coding models compared
Key takeaways Compare axes, not a winner’s list — code quality, context, speed, autonomy, reasoning, cost and openness each pull in different directions. Every provider offers tiers — a frontier flagship, a fast/cheap small model, and open-weight options you can run locally. Read benchmarks skeptically — a leaderboard isn’t your codebase.
This is lesson 7 of the path. The previous lesson mapped who makes the models; this one is about telling them apart in a way that survives the next release. The trap is to want a ranked table — “model A beats model B at coding” — but any such table is wrong within weeks and was never measuring your project anyway. So instead we’ll compare the axes along which coding models differ. By the end you’ll know what “more” of each axis actually buys you, the tiers nearly every provider offers, the difference between specialized and general models, and how to read a benchmark without being fooled. We turn all of this into an actual choice in Choosing a model & provider in Module 6.
The axes that actually distinguish coding models
Think of a model as a point in several dimensions rather than a spot on a ladder. Here are the axes that matter for coding work, and what gaining more of each gives you — and costs you.
| Axis | What it means | What “more” buys you | The catch |
|---|---|---|---|
| Code quality / capability | How correct, idiomatic and complete the code is | Fewer bugs, better designs, harder problems solved | Usually correlates with higher cost and slower responses |
| Context window | How much text (code + history) fits in one request | Whole files or modules in view at once | Big contexts cost more per call and can dilute focus |
| Speed / latency | How fast tokens come back | Snappy completions, tight feedback loops | The fastest models are usually the least capable |
| Autonomy / tool use | How well it runs tools, edits files, chains steps | True agentic work, not just suggestions | More autonomy means more to supervise and verify |
| Reasoning depth | Whether it “thinks” before answering | Better multi-step problem solving and debugging | Reasoning models are slower and pricier per answer |
| Multimodal input | Accepts images/diagrams, not just text | Read a screenshot, a chart, a UI mock | Not every task needs it; adds cost when unused |
| Cost | Price per unit of work | More budget headroom for big or frequent calls | Cheapest is rarely most capable — see Understanding the cost |
| Availability / limits | Rate limits, caps, regional access | Run more, longer, without throttling | Newest/best models often have the tightest limits — see Usage limits & tiers |
| Open vs closed | Hosted proprietary or downloadable open-weight | Control, privacy, offline use (open); frontier capability (closed) | Open-weight usually trails the frontier and needs your hardware |
Two themes run through the table. First, the axes trade against each other — fast, cheap and most-capable rarely coexist in one model, so picking a model is really picking which trade-offs you can live with. Second, a “con” only counts if it touches your task. A tiny context window is a dealbreaker for refactoring a 5,000-line DSP file and a non-issue for autocompleting one function. Judge each axis against what you need, exactly as the language-tour lesson judges programming languages in context rather than crowning a winner.
Reasoning depth vs fast responses
This axis deserves a closer look because it’s where the field has moved most. Some models answer quickly in a single pass. Others — often called reasoning models — spend extra compute generating a hidden chain of intermediate steps before they commit to an answer. For a one-line completion or a quick question, that thinking is wasted latency and cost. For a gnarly bug across several files, or designing an algorithm, the deliberation often pays for itself in a correct answer the fast model would have fumbled. The skill is matching the mode to the moment: fast for routine edits, deep reasoning for genuinely hard problems.
Model tiers: nearly every provider offers the same shape
Whatever the family, providers tend to ship a similar lineup, which makes the landscape easier to reason about.
- A frontier flagship. The most capable (and most expensive, often slowest) model in the family. Reach for it on the hardest reasoning, the trickiest refactors, the unfamiliar code you can’t yet read.
- A fast, cheap small tier. Most families have a lighter model — the “mini / flash / haiku / lite” class. It’s cheaper and quicker, trading some capability for responsiveness. It’s the workhorse for high-volume, lower-stakes work: simple edits, formatting, quick questions, in-editor completion.
- Open-weight local options. Downloadable models you run yourself for privacy, offline use, or cost control on your own hardware — at the price of trailing the frontier and needing capable kit.
A practical pattern falls out of this shape: route by difficulty. Use the small tier by default, escalate to the flagship when a task is genuinely hard, and keep an open-weight option for anything that must stay on your machine. Working on GopherTrunk, you might let a cheap fast model rename symbols and tidy tests, call in the flagship to puzzle out a subtle signal-processing bug, and run a local model when decoding captures you can’t send off-machine. We explore mixing models deliberately in One model vs. many.
Coding-specialized vs general-purpose models
Some models are trained or tuned specifically for code; most strong coding happens on general-purpose frontier models that were trained heavily on code alongside everything else.
General-purpose models tend to win for real development because coding rarely happens in isolation — you also need the model to read documentation, reason about requirements, explain its changes, write the commit message, and drive tools. A model that only emits code is awkward for that. Code-specialized models earn their place in narrower roles: very fast, low-latency in-editor completion (where a small model trained on code shines), or compact open-weight models tuned for code that you can run locally. Start with a capable general model, and reach for a specialist only when a concrete need — raw completion speed, self-hosting, a niche language — justifies it.
How to read benchmarks skeptically
Benchmarks are seductive: a single number, a clean ranking, a clear “winner.” Treat them as a loose signal and keep three cautions in mind.
Benchmark contamination. Models learn from huge swaths of the internet, and popular benchmark problems often end up in the training data. When that happens, a high score can reflect memorisation rather than genuine skill — the model has effectively seen the answer key. A model can ace a famous coding benchmark and still stumble on your unfamiliar codebase.
A leaderboard isn’t your codebase. Benchmarks measure narrow, often self-contained tasks. Your work is sprawling, idiosyncratic, full of local conventions and half-documented context. A model’s score on someone else’s test predicts surprisingly little about how it behaves on yours.
Rankings churn, and small gaps are noise. Positions shuffle with every release, and a one- or two-point difference rarely matters in practice. Chasing the current chart-topper costs more attention than it returns.
The honest test is the boring one: take two or three candidate models, give them real tasks from your actual project, and see which output you trust and ship with least rework. That single experiment tells you more than any leaderboard — and it’s the method we build into Choosing a model & provider.
Quick check: why should you be skeptical of a model's high coding-benchmark score?
Recap
- Compare axes, not a ranking — code quality, context, speed, autonomy, reasoning, multimodality, cost, availability and openness each trade against the others.
- A con only counts in context — a small context window or slow speed matters only if your task needs the opposite.
- Reasoning vs fast — deliberate models pay off on hard problems; fast models win on routine, high-volume edits.
- Common tiers — a frontier flagship, a fast/cheap small tier (“mini/flash/haiku/lite”), and open-weight local options; route by difficulty.
- Specialized vs general — general frontier models suit most real work; reach for code-specialists for speed, self-hosting or niche needs.
- Read benchmarks skeptically — contamination, narrow tasks and constant churn mean a leaderboard isn’t your codebase; test candidates on your own work.
Next up: the four ways a model actually reaches you — the chat app, the API, terminal agents and IDE plugins — in Interfaces: app, API, CLI & IDE.
Frequently asked questions
What makes one coding model better than another?
There’s no single ‘better’ — models differ along several axes: raw code quality, context-window size, speed, how autonomously they can use tools, reasoning depth, multimodal input, cost, availability and whether they’re open or closed. A model that’s excellent on one axis may be middling on another. The right model is the one whose strengths line up with what your task needs.
Should I use a coding-specialized model or a general-purpose one?
Most developers do well with strong general-purpose frontier models, which are heavily trained on code and handle the surrounding reasoning, explanation and tool use that real work needs. Code-specialized models can shine at narrow tasks like fast in-editor completion or running fully offline. Try the general one first; reach for a specialist when a specific need (speed, local hosting, a niche language) justifies it.
Can I trust AI coding leaderboards?
Treat them as a loose signal, not a verdict. Benchmarks can suffer from contamination (the test problems leaked into training data), they measure narrow tasks that may not resemble your codebase, and rankings churn constantly. A leaderboard tells you a model can do well on someone else’s test; only trying it on your own work tells you whether it helps you.