Hardcoding one AI model is already technical debt

Teams running more than a billion tokens a month through Vercel's AI Gateway used an average of 42 different models in September 2026, and nearly half of all teams changed their most-used model that month. That is the number worth reacting to — not the headline that open-weight models now take 71.5% of tokens for 23% of spend. Model choice stopped being an architecture decision and became a runtime one, which means a model id hardcoded in your application is already technical debt.

What the production data actually says

These are not benchmark results. They are anonymized aggregate figures from traffic that really ran, published in Vercel's AI Gateway production index for September 2026.

The New Stack's coverage frames the other half of the same fact: open weight takes the volume while Anthropic still takes 64% of the money.

"Open weight" is not one thing

The aggregate hides a spread wide enough to invalidate the category. Inside that same 71.5%:

• DeepSeek: 58% of tokens for 6.6% of spend.
• Z.ai (GLM): 7.2% of tokens for 4.5% of spend.
• Moonshot (Kimi): 3.6% of tokens for 13.2% of spend — the ratio inverts completely.

So one open-weight model costs roughly an order of magnitude more per token than another. "We moved to open weight to cut costs" is not a strategy; it is a sentence that can be true or false depending on which open-weight model you picked and for which task. The spread between open models is now wider than the gap people imagine between open and closed.

Cheaper tokens produced bigger bills

The two most counterintuitive numbers in the index sit next to each other: average price per token fell 47%, and gateway spend grew 13x over the same period.

This is Jevons paradox arriving in inference. Falling unit cost did not reduce the line item — it removed the constraint that was holding usage down. Every feature that was too expensive to ship at last year's prices became shippable, and teams shipped it.

The planning consequence is specific: do not budget inference as a cost that declines. Model prices fall and your bill grows, because capability per dollar is what actually governs what you build. A founder forecasting flat AI spend because prices are dropping is forecasting the wrong variable.

The coupling that will hurt

Put the two findings together — half of teams change their primary model monthly, and the cost spread inside one category is 10x — and the architectural conclusion writes itself. Anything that makes switching models expensive is now the expensive thing.

What that looks like in code, from what we see taking AI-built software to production:

1. Model ids inline at call sites. A string literal in twelve files is a migration, not a config change. One place, one name, injected.
2. Prompts written to one model's quirks. Formatting tricks tuned to a specific family silently stop working on the next one. Keep prompts behaviour-specified, and test them against at least two models.
3. No task-to-model mapping. Classification, extraction, bulk summarization and hard reasoning have cost profiles that differ by 10x or more. Routing by task class is where the savings actually are — not in picking a cheaper default for everything.
4. No eval harness. Without one, switching models is a leap of faith, so teams don't switch, so they overpay on the tasks that never needed the expensive model. The eval suite is what converts a price drop into a budget cut.
5. Provider SDKs spread through the domain layer. The gateway pattern exists for a reason: one adapter, one place to add a fallback, one place to see spend.

None of this is exotic. It is the ordinary discipline for any dependency that changes faster than your release cycle — and right now, that dependency changes monthly.

What we would build first

If an AI feature is already in production and you have none of the above, the order we would work in: extract the model id to configuration, write evals for your three highest-volume tasks, then route those three by task class. That sequence captures most of the cost benefit before any rewrite, and it turns the next model switch into a configuration change instead of a project.

This is the unglamorous layer between a feature that demos and a system whose economics survive a year — the work we do on AI-built software heading to production. It also rhymes with something we wrote earlier: generation got abundant, judgment got scarce. Model routing is judgment, encoded.

Key takeaways

• Teams processing 1B+ tokens used 42 models on average in September 2026, and about half of all teams changed their most-used model that month. Model choice is a runtime decision now.
• Open-weight models took 71.5% of tokens but only 23% of spend; Anthropic alone still took 64% of the money.
• "Open weight" is not one price point: DeepSeek ran 58% of tokens for 6.6% of spend while Moonshot ran 3.6% of tokens for 13.2%.
• Price per token fell 47% while total spend grew 13x — cheaper inference expands usage instead of shrinking budgets.
• The expensive coupling is anything that makes switching hard: inline model ids, model-specific prompts, no evals, no task-to-model routing.

FAQ

Should we move to open-weight models to cut costs?

Only with a task-to-model mapping and evals. Open weight took 71.5% of tokens for 23% of spend in aggregate, but within that category one model ran 58% of tokens for 6.6% of spend while another ran 3.6% for 13.2%. The category tells you nothing; the pairing of task to model is what determines cost.

Why did spend grow when prices fell?

Because price was the constraint on usage. Average price per token fell 47% between December 2026 and September 2026 while gateway spend grew 13x. Cheaper inference makes previously uneconomic features shippable, and teams ship them.

How often do teams actually change models?

In September 2026, nearly half of teams changed their most-used model and one in three switched lab entirely — an index record. That is the rate your architecture has to absorb.

What is the first thing to fix in an existing codebase?

Move the model id out of call sites into configuration. It is a small change that converts every future switch from a code migration into a deploy, and it costs almost nothing to do now.

Do we need a gateway product for this?

Not a product necessarily — but you do need the pattern: one adapter between your domain code and any provider SDK, so routing, fallback and spend visibility have somewhere to live.

Sources: Vercel AI Gateway production index, October 2026 · The New Stack