GPT-4-class inference cost roughly $20 per million tokens in late 2022. Today the same performance costs around $0.40 - a 50x reduction. Uber burned through its entire 2026 AI coding budget in four months.
Those two facts are not in tension. They are cause and effect. Understanding why is the most useful thing an engineering or product leader can do right now, because the mental model most teams are using for AI inference cost is quietly wrong.
The number on the pricing page is not your cost
The bill is not: (tokens per request) × (price per token) × (number of requests).
That equation held when AI meant a chatbot answering one question at a time. It breaks the moment you add agents, reasoning steps, or RAG pipelines.
Agentic AI systems - where models take sequences of actions, retrieve information, and call tools autonomously - consume five to thirty times more tokens per task than a simple chatbot.
A simple chatbot query triggers one inference call, but an agentic workflow that calls external tools, verifies outputs, and self-corrects can trigger 10 to 20 model calls for a single user-initiated task.
That changes the business model: the relevant unit is no longer cost per prompt, but cost per completed task.
Reasoning models compound this further. A response with 200 visible tokens might consume 10,000-30,000 total tokens during reasoning. This internal token consumption gets billed but remains invisible in the output, creating situations where per-token pricing appears low but total costs are high.
Some reasoning queries consume over 600 tokens to generate two-word answers.
Then there is the context window problem. MCP tool metadata alone can consume 40-50% of context windows.
Hidden costs - retrieval augmentation, embedding generation, context window management, retry logic - routinely add 40-60% on top of the raw inference bill that most teams are actually tracking.
The result: analysis of 2.4 billion enterprise API calls shows the blended cost of AI dropped 67% year-over-year, from $18.40 to $6.07 per million tokens between Q1 2025 and Q1 2026. That number is accurate. It is also almost entirely irrelevant to what is actually happening to enterprise AI invoices right now.
Jevons' paradox, applied to GPUs
Economists have a name for this. As AI becomes more efficient and the price of a single token drops, total spending tends to rise. Economists refer to this as Jevons' paradox, and it appears clearly in enterprise AI spend.
When tokens get cheaper, token-heavy applications that were previously too expensive become financially viable, so companies run more of them. The added volume outpaces the lower price per token, driving total spend higher even as unit cost drops.
The macro numbers confirm it. The inference price of LLMs has fallen dramatically in recent years. Researchers looked at the results of state-of-the-art models on six benchmarks over three years and measured how quickly the price to achieve those milestones fell. The price to achieve GPT-4's performance on a set of PhD-level science questions fell by 40x per year. The rate of decline varies dramatically depending on the performance milestone, ranging from 9x to 900x per year.
Yet while per-token cost fell nearly an order of magnitude in two years, enterprise consumption rose by more than 100x.
And there is a ceiling problem on top of the volume problem. Flagship reasoning models show near-zero price decline due to a reasoning premium averaging 31.5x non-reasoning prices. The cheap tokens are the old, commoditized capability. The tokens that actually move the needle on hard tasks - multi-step code generation, complex analysis, long agentic runs - are still expensive, and getting more expensive to run as reasoning chains grow longer.
Agentic models require between 5-30 times more tokens per task than a standard GenAI chatbot. While lower token unit costs will enable more advanced capabilities, these advancements will drive disproportionately higher token demand. As token consumption rises faster than token costs fall, overall inference costs are expected to increase.
Gartner's framing is blunt: "Chief Product Officers should not confuse the deflation of commodity tokens with the democratization of frontier reasoning. As commoditized intelligence trends toward near-zero cost, the compute and systems needed to support advanced reasoning remain scarce."
What the inference cost breakdown actually looks like
It helps to be concrete. Frontier models run $2-$15 per million input tokens and $10-$75 per million output tokens. Standard models run $0.50-$5 input and $2.50-$25 output. Smaller models start at $0.10-$1 input.
In production, most organizations spend far more on inference than on training. Industry analyses put the split at roughly 80% of AI budget on inference and 20% on training.
In 2026, inference accounts for approximately two-thirds of all AI compute demand, up from roughly one-third in 2023.
The practical implication: if you are running a coding agent on a frontier model, you are not paying $3 per million tokens for the Slack message that triggered the run. You are paying for the system prompt, the full file context, every tool call's schema, every intermediate reasoning step, and every retry when the first attempt fails. A 128K-token context can cost roughly 64x more to process than an 8K context; output tokens are typically priced 3-5x higher than input.
Four moves that actually reduce the bill
Most AI cost advice collapses into "use a smaller model." That is too coarse to be useful. The real levers are:
Route by task complexity, not by model preference. For multi-agent systems, a hierarchical architecture - budget models for worker agents, frontier models only for the lead orchestrator - can achieve 97.7% of full-frontier accuracy at roughly 61% of the cost. The question to ask before any agent call is: does this step require reasoning, or just retrieval and formatting?
Audit your tool lists. MCP tool metadata can consume 40-50% of context windows. CLI-first or skills-based approaches for production workloads where tool discovery is not needed at runtime can reclaim that capacity. If your agent loads 30 tool definitions into every prompt and uses two of them, you are paying for 28 tools you did not need.
Cache system prompts aggressively. Prompt caching offers 50-90% reduction on repeated context. Every agent that runs in a loop with the same system prompt and tool definitions is a candidate. A teammate like Beagle, which runs in Slack with the same routing context across many requests, treats caching as a first-class concern rather than an optimization to get to later.
Track cost per task, not cost per token. Stanford's Cost-of-Pass framework measures the expected monetary cost of generating a correct solution, not just the cost of generating tokens. The finding: lightweight models are most cost-effective for basic quantitative tasks, large models for knowledge-intensive ones, and reasoning models for complex quantitative problems - despite higher per-token costs. A reasoning model that solves a problem on the first attempt can be cheaper than a cheap model that retries four times.
What the trajectory actually looks like
The direction of travel is not ambiguous. In late 2022, running a GPT-4-class model cost approximately $20 per million tokens. In early 2026, equivalent performance costs $0.40 per million tokens - or less. That is a 1,000x reduction in just over three years, one of the fastest cost declines in computing history.
Recent MIT research found that infrastructure and algorithmic efficiencies are reducing inference costs for frontier-level performance by up to 10x annually.
By 2030, performing inference on an LLM with one trillion parameters will cost providers over 90% less than it did in 2025, according to Gartner.
But the Jevons problem does not resolve on its own. With consumers and enterprises adopting AI agents, token consumption is expected to multiply 24 times, to 120 quadrillion tokens per month, between 2026 and 2030. Cheaper GPUs do not reduce how many times your agent calls the model before it finishes a task.
Inference cost is the line that decides whether your AI product makes money or just makes demos. Track it as a percentage of revenue and aim under 25-30% at maturity.
The teams getting this right are not the ones waiting for prices to fall far enough to make their architecture work. They are the ones who sized their model choice to their task, kept their context lean, and measured in outcomes rather than tokens.