Uber's CTO burned the company's entire 2026 AI coding budget in four months. Not because the tokens were expensive - they weren't. Claude Code adoption among Uber's 5,000-engineer org jumped from 32% to 84% between December 2025 and March 2026, and monthly API costs per engineer were running between $500 and $2,000 . The per-token price wasn't the problem. The problem was that nobody had modeled what an agentic workflow actually consumes.
This is the inference cost story most teams are getting wrong. Inference costs have declined dramatically - 10x to 100x - from 2023 to 2026, driven by model efficiency improvements, hardware advances, and competitive pricing pressure.
GPT-4 launched at $30 per million tokens in 2023; today a frontier-class reasoning model runs through DeepSeek V4 for $0.10 per million. That is a genuine, structural decline. And it is almost entirely beside the point for teams running agents.
What agentic workloads actually cost
The standard framing - "AI is getting cheaper" - describes the price of a single token without accounting for how many tokens a workflow actually burns. A simple chatbot query triggers one inference call, but an agentic workflow - where the agent calls external tools, verifies outputs, and self-corrects - can trigger 10 to 20 model calls for a single user-initiated task.
In 2023, a simple linear workflow (input, retrieval, response) cost $0.04 per interaction. In 2026, a more complex orchestrated system involving tools, reasoning, and iterative loops costs $1.20 per interaction - about 30 times higher. Token prices went down. Task prices went up.
The compounding factors are not always obvious:
Context drift. Agentic systems routinely grow from 200 tokens per request at launch to 10,000+ tokens per request within weeks, as conversation history, tool output accumulation, and system prompt expansion compound silently.
RAG overhead. RAG systems add 2,000-10,000 tokens per query depending on retrieval chunk size and top-k settings. Teams that deploy RAG without tuning these parameters routinely discover their retrieval overhead exceeds their reasoning costs by 3:1.
Background inference. Monitoring agents, document watchers, and compliance surveillance systems run continuously, consuming tokens against every event and data update they process regardless of whether any user requested a response. These workloads were minimal in most 2024 enterprise deployments but represent a meaningful and rising share of the monthly inference bill in 2026.
The net result: per-token costs are falling 10x while aggregate token spend is rising 3-5x.
73% of enterprises exceeded their original AI cost projections last fiscal year.
The metric that actually matters: cost per task
Ask most enterprise leaders about AI costs and they will cite a figure per million tokens. Ask what that spend delivers and the answer becomes far less clear. That gap is not a reporting issue; it is a measurement flaw. Cost per token shows the price of raw input, not the value of what is created.
In 2026, cost per token is no longer sufficient as the primary enterprise AI KPI. Organizations scaling AI with discipline have shifted to a different unit of measurement: cost per workflow.
What does cost per workflow require in practice? Three things:
Task profiling before deployment. Measure the average token-per-task ratio across your specific agent workflows before you commit to a model or a budget. A support-triage agent and a coding agent look nothing alike on a token consumption chart.
Observability at the task level, not just the API level. AI FinOps must govern behavior - how often agents call expensive models, how much context they include per call, whether they cache or repeat identical requests, how many parallel agent threads are spawned. This is a fundamentally different observability and control problem.
Budget tied to tasks completed, not tokens purchased. If your AI coding budget is expressed in millions of tokens, you have no lever that maps to engineer output. If it is expressed in pull requests reviewed or issues resolved, you do.
Model routing: the practical lever
Model routing - sending each request to the cheapest model that can handle it, rather than defaulting every call to a frontier model - is the most actionable way to close the gap between per-token prices and per-task costs.
ICLR 2025 work cut cost 85% on MT Bench while keeping 95% of GPT-4 Turbo quality, with a matrix-factorization router sending only 14% of queries to the strong model. That number is not from a toy benchmark. Research from LMSYS shows that a well-trained router can achieve 95% of GPT-5.2 quality while using it for only 26% of requests.
The arithmetic is straightforward. Routing 70% of traffic to a $0.10/million token model and 30% to a $3/million token model yields a blended cost of $0.97/million - dramatically lower than using the expensive model for everything.
For a Slack or Teams integration handling a mix of lookup questions, summarization requests, and occasional multi-step reasoning, the tiers might look like this:
| Task type | Example | Model tier | Why |
|---|---|---|---|
| Lookup / classification | "What's our PTO policy?" | Small (3-8B) | Single-turn, well-scoped, no reasoning chain needed |
| Summarization | Condense a 2,000-word thread | Small-mid (8-14B) | Bounded input, low error cost |
| Multi-step agent task | Research competitors and draft a brief | Frontier | Requires planning, tool calls, judgment |
| Code review | Scan a PR for regressions | Mid-frontier | Reasoning matters; errors have real costs |
Organizations running a tiered model architecture achieved a median blended cost of $2.31 per million tokens in Q1 2026. Organizations routing every workload to frontier models paid $18.40 per million tokens. That 8x difference compresses fast when you're running an agent across a 50-person team all day.
A practical routing stack in 2026 is not exotic. An LLM gateway - a layer between your app and model providers that centralizes routing, caching, failover, cost tracking, and governance behind a single API - makes this manageable. Common production gateways include LiteLLM, OpenRouter, Cloudflare AI Gateway, Kong AI Gateway, and Bifrost.
Beagle in action#product-team, 10:41amWhat to actually do this week
The Uber story is instructive not because the spend was reckless but because it was invisible. The fix is not to throttle usage - agents that work are worth paying for. The fix is to know what you are paying per task so you can tell which tasks are worth it.
A concrete starting checklist:
- Tag every agent call with the task type in your logging layer, not just the model name. You cannot route what you cannot classify.
- Set a context budget per task type. Unbounded RAG retrieval is the single biggest silent cost driver. A ceiling of 4,000 tokens for retrieval chunks on standard lookups is a reasonable starting constraint.
- Run a two-week cost-per-task audit before scaling any agent to the full team. What a 5-person pilot consumes scales non-linearly once calendar integrations, background watchers, and retry logic turn on at team size.
- Route lookup tasks to a small model by default. SLMs (sub-27B models) can substitute large models in roughly 80% of agent subtasks. That ratio holds particularly well for Slack-style tasks: summarization, Q&A over a known corpus, classification, and short-form drafting.
An AI teammate like Beagle is already built around this kind of task-aware routing - matching the model to the job rather than sending every thread lookup through the same inference stack as a complex multi-step workflow.
Agentic AI inference cost: common questions
Why are AI bills rising when per-token prices are falling?
The cost problem in agentic AI is not price per token. It is the explosion in tokens per task. Agentic workflows trigger 10-20 model calls per user-initiated task, resend accumulated context on each call, and run background agents continuously - none of which appeared in 2024 budget models.
What is cost per workflow and why does it replace cost per token?
Cost per workflow is the total inference spend to complete one unit of work - a ticket resolved, a PR reviewed, a brief drafted. It accounts for every model call, retrieval chunk, and retry in a task, making it the only metric that connects AI spend to business output. Cost per token cannot do this because the same token price buys wildly different amounts of useful work depending on how the system is designed.
How much does model routing actually save?
Model routing cuts real LLM bills 40-85% with no visible quality loss in controlled evaluations. In production, savings depend on workload mix. A Slack integration handling mostly lookups and summaries can route most traffic to small models; a coding agent with complex multi-file reasoning cannot. Measure your own task distribution before projecting savings.
Which tasks can small models handle reliably?
Single-turn lookups, summarization of bounded documents, intent classification, and short-form drafting are all well-handled by 3-14B models at current capability levels. A 3.8B model like Phi-4-mini achieves a similar level of multilingual language understanding and reasoning ability to much larger models on narrow, well-defined tasks. Multi-step planning, tool chaining across ambiguous inputs, and complex code review still benefit from frontier models.
How do I start tracking cost per task without a big observability build?
Start with logging. Tag every LLM call with a task type and a trigger source before you add any routing logic. Two weeks of tagged logs will show you where cost concentrates. In most teams, 20% of task types account for 80% of spend - and that 20% is usually where a routing rule pays for itself in a week.