GPT-4-class performance now costs roughly $0.40 per million tokens - down from $20 in late 2022. Most teams read that and assume their AI spend is under control. Their invoices disagree.
The disconnect is not a pricing trick. It is a unit-of-measurement problem. You are watching cost per token fall while the number of tokens your workflows consume rises faster. The line your CFO sees is the product of those two forces - and right now, volume is winning.
Why cheaper tokens produce higher bills
Token prices have fallen about 10x every year since 2021 - equivalent GPT-4 class performance that cost $20 per million tokens in late 2022 now runs closer to $0.40. That is a real efficiency gain. But total AI spend for companies in production has gone up over the same period, because more capable models invite more ambitious use cases, which means more tokens, more calls, and more infrastructure complexity.
Economists have a name for this pattern. The Jevons Paradox - first observed with coal - says that when a resource becomes more efficient to use, consumption rises so much that total usage increases even as unit cost drops. Applied to AI tokens, use cases that were too expensive to justify last year are worth running once tokens are cheap, so more workflows move into production.
The mechanism is concrete. A 2024 AI interaction was simple: a user submits a prompt, the model responds, consuming roughly two thousand tokens. A 2026 agentic workflow - where an orchestrator decomposes a task, selects tools, calls sub-agents, validates outputs, and retries on failure - looks nothing like that. Every one of those steps burns tokens.
Agentic AI systems, where models take sequences of actions, retrieve information, and call tools autonomously, consume five to thirty times more tokens per task than a simple chatbot. A ReAct trace with tool calls compounds this further: in a five-agent system with fifty reasoning steps operating on an 8,192-token planning document, context broadcasting alone generates over two million tokens of overhead, most of it unchanged context retransmitted at every synchronization boundary.
The routing decision that accounts for most of the gap
A Q1 2026 analysis of 2.4 billion enterprise API calls found that organizations running a tiered model architecture achieved a median blended cost of $2.31 per million tokens. Organizations routing every workload to frontier models paid $18.40 per million tokens. That 87% gap is the direct financial consequence of one architectural decision made at the start of the deployment process, and in most cases never explicitly revisited.
The mismatch is systematic. Enterprise data from AI infrastructure platforms shows that in early 2025, roughly 73% of enterprise token volume was being routed to the two most expensive model tiers. A standard FAQ response that could run on a model priced at $0.04 per million tokens was instead running on a reasoning engine priced at $180 per million - a 4,500-fold cost multiplier for identical quality output.
Models vary in cost per token by 10x or more. Frontier models are appropriate for tasks that genuinely require state-of-the-art reasoning - but a claims-processing agent that extracts data from a standard form and checks it against policy terms does not need the same model that powers a research assistant parsing thousands of documents.
The open-weight case makes this even sharper. In many cases, the most cost-effective option is an open-source model - with proper tuning and guardrails, open-source models can perform at or near frontier benchmarks for many enterprise tasks while reducing costs by 40 to 90%.
Beagle in action#eng-ops, Thursday afternoonWhat the open-weight model line actually changes
Alibaba's Qwen3 family is worth understanding specifically in this context, not as a benchmark trophy, but as a cost lever.
The small MoE model Qwen3-30B-A3B outcompetes QwQ-32B with 10 times the activated parameters, and even Qwen3-4B can rival the performance of Qwen2.5-72B-Instruct. Alibaba open-weighted two MoE models: Qwen3-235B-A22B, with 235 billion total parameters and 22 billion activated, and Qwen3-30B-A3B, with 30 billion total parameters and 3 billion activated.
The activated-parameter count is the inference cost number that matters. While Qwen2.5 was pre-trained on 18 trillion tokens, Qwen3 uses nearly twice that amount - approximately 36 trillion tokens covering 119 languages and dialects. More training data baked into fewer active parameters at inference time is exactly the trade that makes self-hosted open-weight models economically interesting.
The big architectural story is the hybrid thinking mode. Every Qwen3 model ships with a toggleable "thinking" behavior - enable it, the model produces a chain-of-thought before its final answer, similar to how DeepSeek R1 works. Disable it, the model responds directly with no reasoning trace. This matters for inference economics because reasoning models that think step by step can require 5-30× the compute of a standard response - that level of reasoning suits complex architectural decisions, not autocomplete. One model that lets you make this choice per-query - rather than routing to two separate models - reduces orchestration overhead.
For teams hitting the limits of hosted APIs, Qwen3-32B and Llama 4 Scout are essentially tied at $0.78-$0.83 per million tokens on comparable hardware (both run on a single H100). That is the self-hosted comparison point: a single-GPU model with near-frontier performance on many tasks, at a token rate well below the mid-tier hosted market.
The honest caveat: several organizations cannot use Qwen and other Chinese open models for branding or compliance reasons - people vastly underestimate the number of companies that cannot use Qwen and DeepSeek open models because they come from China. That eliminates the option for a meaningful share of enterprise teams regardless of the economics.
How to actually manage inference cost at the task level
The right unit to track is tokens per task per dollar - the same way SaaS teams track customer acquisition cost. Build a unit economics model where the basic metric is tokens-per-task-per-dollar, and track it monthly. If you are not budgeting at the tokens-per-task level, your forecasts may be off by an order of magnitude.
Concrete levers, in rough order of impact:
Model routing. Classify each request and route it to the cheapest model that can handle it. Classification, intent detection, and structured extraction rarely need frontier capability.
Prompt caching. Reusing a 3,000-token system prompt across 10,000 daily requests with prompt caching avoids paying for 30 million redundant input tokens per day.
Thinking budget control. For hybrid reasoning models like Qwen3 or Claude's thinking mode, cap or disable the reasoning trace on queries where it adds no value. A Slack lookup does not need chain-of-thought.
Output length discipline. Controlling output length is the single most effective cost optimization. A prompt that generates 500 tokens of output costs 2-5x less than one that generates 2,000 tokens, even with identical input.
Retry budget. When an output does not meet validation criteria, the agent resubmits with the full conversation history resent as context. An agent running ten correction cycles can consume fifty times the tokens of a single linear pass. Set explicit retry limits before you touch anything else.
Price-performance moves week to week as new models ship. Research from Epoch AI found that the price to reach a fixed performance level has fallen between nine times and 900 times per year, depending on the benchmark - which means the cheapest model for your task in January is rarely the cheapest by April. The practical consequence: revisit your routing logic quarterly, not once at deployment.
AI inference cost per task: common questions
What is the difference between cost per token and cost per task?
Cost per token is the rate a provider charges per unit of text. Cost per task is what you actually pay for a complete workflow - a query answered, a ticket triaged, a document summarized. Agentic tasks can consume 1-3.5 million tokens each through retries and tool calls. The per-token price tells you almost nothing about total spend without the per-task token count.
Why do enterprise AI bills keep rising when token prices are falling?
Token volume is growing faster than prices are falling. Despite token prices falling by 280 times over two years, total enterprise AI spend has risen by 320% over the same period. Cheaper tokens unlock more ambitious use cases, which consume more tokens - a textbook Jevons Paradox.
How much does a reasoning model cost compared to a standard model?
Reasoning models are 5-10x more expensive than standard models because they use much more compute per query through extended thinking time. On complex math problems, a model like o3 can generate 25,000 reasoning tokens for a 500-token visible answer - a 50x difference in effective cost compared to a simple query.
Can open-weight models meaningfully cut inference costs?
Yes, for many enterprise tasks. For high-volume production workloads where cost per completion drives unit economics, mid-tier open-source models are the clear choice. DeepSeek V3.2, Kimi K2, and Qwen3-235B all sit in a range where you get near-frontier quality without the frontier price tag. The constraint is compliance: regulated industries and teams with data-residency requirements may not be able to use models from Chinese labs.
What is a realistic target for AI inference cost per user per month?
The $5-$15 per user per month post-optimization target and $50-$100+ per user pre-optimization range are representative of 2025-2026 pricing levels. Absolute numbers will decrease as model pricing continues to deflate, but the optimization ratios remain stable. The 5-10x gap between optimized and unoptimized is a consistent pattern across team sizes and use cases - and it is almost entirely a routing and caching problem, not a hardware one.