Why Is Your AI Inference Cost Rising When Tokens Are Cheaper?

Token prices have fallen about 10x every year since 2021. Equivalent GPT-4-class performance that cost $20 per million tokens in late 2022 runs closer to $0.40 today. And yet, if you are running AI in production right now, your monthly bill is almost certainly higher than it was a year ago. Total AI spend for companies in production has gone up, not down. More capable models invite more ambitious use cases, which means more tokens, more calls, and more infrastructure complexity.

The two facts coexist, and understanding exactly why they do is the most practically useful thing you can read about AI economics in 2026.

The token price drop is real - the savings are not

According to Stanford University's 2025 AI Index Report, the inference cost for a system performing at the level of GPT-3.5 dropped over 280-fold between November 2022 and October 2024. That number gets cited everywhere, and it is true. It is also almost completely irrelevant to what shows up on most teams' invoices.

The reason is straightforward. Reasoning models are 5-10x more expensive than standard models because they use much more compute per query through extended thinking time. And the newest frontier tier is not getting cheaper at all. OpenAI raised GPT-5.5 API pricing to $5 per million input tokens and $30 per million output tokens on April 23, 2026 - roughly double the GPT-5.4 pricing of $2.50/$15. OpenAI's justification was that the model is more token-efficient, so you should need fewer tokens per task. Users report total bill increases of around 40 percent, even as the model's token efficiency improved.

Flagship reasoning models show near-zero price decline due to a reasoning premium averaging 31.5x non-reasoning prices. Meanwhile, closed-frontier providers (Claude, ChatGPT, Gemini) all started November 2023 between $7 and $30 per million tokens, fell over two years, then reversed in 2026. GPT-5.5 launched at $5/$30 per million tokens, doubling GPT-5.4 on both sides.

The comfortable narrative - that AI gets cheaper with every model generation - held through 2025. Then 2026 happened. Closed-frontier providers fell over two years and reversed. The price floor is back up, driven by a genuine compute capacity crunch at the labs.

Agents are the real multiplier

Even if frontier model prices had kept declining, agentic workflows would have eaten the savings.

A simple chatbot query triggers one LLM inference call. An agentic workflow - where an autonomous AI agent reasons iteratively, breaks down a task, calls tools, verifies outputs, and self-corrects - may trigger 10 to 20 LLM calls to complete a single user-initiated task.

A single agentic workflow execution - retrieving context, reasoning through steps, calling tools, validating outputs - can consume 15,000-80,000 tokens per task completion, compared to 500-2,000 tokens for a simple Q&A interaction.

That is not a marginal difference. At scale, it rewrites the budget entirely.

Uber's CTO said something every engineering leader should sit with: "I'm back to the drawing board, because the budget I thought I would need is blown away already." Claude Code adoption jumped from 32% to 84% of Uber's 5,000-engineer org between December 2025 and March 2026. By April, the entire annual AI budget was gone. Monthly API costs per engineer were running between $500 and $2,000.

The pilot economics, calculated on single-query API calls, bore no relationship to the production economics of multi-step agentic loops running thousands of times per day. Almost every team discovers this the same way: through a production bill that looks nothing like the test environment.

Why "just use a cheaper model" is not as simple as it sounds

The obvious fix is routing: send simple tasks to small, cheap models and reserve the frontier tier for work that actually needs it. That is correct. A routing layer that classifies incoming queries by complexity and directs simple tasks - summarisation, classification, extraction, formatting - to cost-optimised models, while reserving frontier models for complex reasoning, can divert 80% of routine traffic to cheaper tiers with minimal quality loss.

But routing requires knowing which tasks are actually simple. Most teams do not know this until they have run evals against their own workloads. For 80-90% of use cases, open-weight models match or exceed frontier mini-tier capabilities on many benchmarks. Edge cases with complex reasoning, multimodal processing, or domain-specific tasks still favor frontier models.

DeepSeek V3.2, Kimi K2, and Qwen3-235B all sit in a range where you get near-frontier quality without the frontier price tag.

The 10x difference in price from Kimi K2.6 to GPT-5.5 does not mean the model quality differs by 10x. Kimi K2.6 leads SWE-Bench Pro at 58.6%, beating GPT-5.4 at 57.7% and Opus 4.6 at 53.4%.

There is a real cost to switching that gets underweighted in these comparisons: the harder problem is that the model you license is not the same week to week. While the model weights are stable, it's the harness around the model - default reasoning effort, system prompts, tool-call routing, cache layers - that's changing. Open-weight models do not have this problem. The weights you download are the weights you run, indefinitely, wherever you want.

What actually controls your inference bill

There are four levers, and most teams only pull one.

Model choice is the obvious one, but it is downstream of the others. Prompt caching can cut costs 50-90% on repeated context. Batching cuts costs around 50% for delayed responses. Routing easy queries to cheaper models is the most impactful structural change.

Observability is where most teams are weakest. Average monthly AI spend reached $62,964 in 2024, and only 51% of organisations said they could confidently evaluate the ROI of that spend. That gap between what organisations spend and what they can explain is an inference cost visibility problem. Without attribution - knowing which workflow, which prompt, which agent step generated which cost - you cannot route intelligently, you can only guess.

Agentic loop design matters more than model price at scale. Querying the agent's memory layer before making any expensive LLM call is a powerful optimisation. If a semantically similar problem was solved previously and the plan is stored in a vector index, retrieve and reuse rather than re-reason. This reduces latency from 30 seconds to 300ms for cache hits, and cost to near zero.

Prompt caching and batching are still underused. Cached-input reads drop 90% on supported GPT models, which matters a lot for repeated system prompts and long-context agents. A teammate like Beagle, which runs inside Slack and operates on a draft-and-approve model, is well-positioned to take advantage of this: the system prompt and channel context are stable across most requests, making cached prefills a natural fit.

The structural truth about frontier AI pricing

For some benchmarks, roughly half of measured progress is associated with increasing inference prices: holding benchmark cost fixed, performance improves at about half the rate implied by the unconditional time trend. Smarter models cost more to run - not just because they are larger, but because the harder tasks we put them on require more tokens to complete.

If an enterprise wants to upgrade from a generative AI chatbot to an agentic assistant, "it's not just that the personal assistant is doing more queries, it's that every single query costs five to 30 times more tokens."

Applications that are uneconomical today may become economical next year. Build for the trajectory, not the snapshot. That is true for the commodity tiers. For frontier-tier reasoning, the trajectory reversed in Q1 2026, and the labs' financial pressures - OpenAI is projected to lose $14 billion in 2026, Anthropic a reported $11 billion

• suggest those pressures are structural, not temporary.

The practical answer is not to avoid frontier models. It is to stop using them as a default and start using them as a deliberate escalation. Run evals, build a routing layer, cache aggressively, and revisit your model choices every quarter. The per-token price will keep moving. Your job is to make sure your bill moves more slowly than the capability does.