Uber's 5,000-person engineering org adopted Claude Code fast - from 32% of engineers in December 2025 to 84% by March 2026. By April, the entire annual AI budget was gone.
Monthly API costs per engineer were running between $500 and $2,000. This was not runaway spending. It was a measurement problem: the team was tracking cost per token, and the relevant unit had already changed.
Why token price is the wrong number to watch
The relevant unit is no longer cost per prompt, but cost per completed task. That shift matters because the two numbers can diverge by orders of magnitude.
When GPT-4 launched in March 2023, it cost $30 per million input tokens. By mid-2026, you can get equal-or-better quality from open-weight models for under $0.50 per million tokens. That curve looks like a gift. But the most confusing aspect of the 2026 situation for enterprise finance teams is the simultaneous reality of falling unit costs and rising total bills. Per-token inference prices have fallen between 9x and 900x per year for various performance milestones - yet the same enterprises watching token prices collapse are seeing their monthly AI bills multiply.
The mechanism is simple once you see it. A simple chatbot query triggers one inference call, but an agentic workflow - where an agent calls external tools, verifies outputs, and self-corrects - can trigger 10 to 20 model calls for a single user-initiated task.
A single agentic workflow execution - retrieving context, reasoning through steps, calling tools, validating outputs - can consume 15,000-80,000 tokens per task completion, compared to 500-2,000 tokens for a simple Q&A interaction.
Research from Microsoft and Stanford's Digital Economy Lab found that agentic tasks consume roughly 1,000 times more tokens than standard chat interactions. Cut the token price by 75%, deploy a system that consumes 250 times more tokens per task, and the math is unambiguous: you pay more.
What OpenAI and Gartner said in the last four weeks
Two things happened in late June and early July that change the picture for teams planning AI budgets.
The first was OpenAI's Jalapeño announcement. On June 24, OpenAI unveiled its first custom-built inference processor, designed and manufactured in collaboration with Broadcom and named Jalapeño.
It went from initial design to manufacturing tape-out in just nine months
- an unusually fast ASIC development cycle. The companies are aiming for initial deployment of the Jalapeño chips by the end of 2026.
Jalapeño is specifically designed for inference, the process of running pre-built AI models in response to user commands, with OpenAI emphasizing the chip's low operating cost when running real-time coding models.
Separately, OpenAI engineers reportedly told colleagues in June 2026 that they had found a software-based optimization capable of cutting the inference cost of some existing models by more than half, according to reporting first surfaced by The Information.
Method details, rollout timing, and customer-facing price changes remain unconfirmed publicly, so the claim is still an internal efficiency target rather than a launched discount.
Then on July 16, Gartner published a forecast worth reading carefully. By 2030, performing inference on a large language model with one trillion parameters will cost GenAI providers over 90% less than it did in 2025. The buried clause is the one that matters for teams: while lower token unit costs will enable more advanced GenAI capabilities, these advancements will drive disproportionately higher token demand. As token consumption rises faster than token costs fall, overall inference costs are expected to increase.
Read those two facts together. Infrastructure gets dramatically cheaper. Bills go up anyway. Microsoft CEO Satya Nadella acknowledged the dynamic directly when DeepSeek released its low-cost model: "Jevons Paradox strikes again." Lower unit costs are an invitation to consume more, not a mechanism for spending less.
The non-obvious implication for real teams: cheaper inference from Jalapeño will not arrive as a discount on your current usage pattern. It will arrive as permission to run more agents, on longer tasks, against bigger context windows. Your bill may stay flat or climb. The only lever you control is how efficiently each agent completes its task.
Three things that actually reduce cost per task
These are engineering decisions, not pricing negotiations.
Route tasks by complexity, not habit. Routing 80% of routine inference traffic to cost-optimized models while reserving frontier models for complex tasks reduces inference spend by 60-80% with minimal quality impact.
Routing all steps through a frontier model charges frontier prices for commodity-grade work. Production teams achieving 90% token cost reductions are routing structured extraction, simple classification, and tool schema parsing to smaller open-weight models - Qwen3, Llama 3 70B, Mistral - while reserving frontier models for planning, reasoning, and user-facing generation. A teammate like Beagle, which handles Slack lookups and short Q&A, should almost never be touching a frontier model for those calls.
Watch context accumulation. A twenty-step ReAct trace with tool outputs can easily hit 50,000 tokens of accumulated context. At that point, the model spends more tokens reading its own history than reasoning about the current step.
Context compression can reduce LLM costs by 50% in production agentic pipelines - without touching model selection.
Set hard budget guardrails before you need them. No major agent framework ships with a native dollar-denominated budget cap. LangChain, CrewAI, and AutoGen all expose token counting as a post-hoc observable, not a pre-execution policy. One fintech team discovered this the hard way: two LangChain agents entered an infinite conversation cycle, and neither a human nor a monitoring system caught it for 11 days. The final bill was $47,000 in inference costs from a pipeline budgeted at under $200 per month.
The fix is instrumentation first. Tag every LLM call with agent ID, task type, model used, and token counts. Route telemetry to a cost tracking system so you can identify which agents, workflows, or users are responsible for cost spikes.
Beagle in action#engineering, Monday morningWhat this means if you're evaluating or expanding AI at work
The table below maps task type to the inference strategy that fits it. The point is not to memorize tiers - it's to stop treating every call as equivalent.
| Task type | Token demand | Recommended tier | Why |
|---|---|---|---|
| Simple lookup / Q&A | ~500-2,000 | Small or mid model | One call, short context, clear answer |
| RAG over docs | ~2,000-10,000 | Mid model + cache | Retrieval overhead is the cost, not reasoning |
| Multi-step agent (research, triage) | ~15,000-80,000 | Tiered routing | Break into sub-tasks; route by step complexity |
| Long-running planning agent | 50,000+ | Frontier + hard budget cap | Necessary, but needs a kill switch |
SLMs - sub-27B models - can substitute large models in 80% of agent subtasks. That one number should change how you scope your next AI deployment. If you are routing everything through GPT-5.6 or Claude Opus because it feels safer, you are most likely leaving 60-80% of your inference budget on the table.
One team cut monthly API costs from $40,000 to $24,000 after auditing token usage and routing simpler subtasks to cheaper models. No product changes. Just routing discipline.
The infrastructure news of the last few weeks - Jalapeño, the software optimization, the Gartner forecast - all point in the same direction: inference will get cheaper at the hardware layer, and usage will expand to fill the savings. The economics of AI may be shifting from "who can buy the most GPUs" to "who can keep those GPUs busiest." For teams, the equivalent shift is from "which model should we use" to "which task actually needs this model." That second question is worth asking before your next budget cycle, not after.
AI inference cost for teams: common questions
What is the agentic inference multiplier?
According to Gartner's March 2026 analysis, agentic workflows require 5 to 30 times more tokens per task than a standard chatbot query, because they make multiple model calls per user-initiated task. A simple chat reply might cost $0.001; a multi-step agent completing the same request can run $0.10 to $1.00.
Why is my AI bill going up if token prices are falling?
Token prices have dropped roughly 95% since 2023, but usage volume is growing faster than prices are falling. Each agentic workflow triggers 10-20 model calls, RAG architectures inflate context windows, and always-on agents run 24/7. Total spend rises even as the per-token cost shrinks.
What did OpenAI announce about inference costs in June 2026?
OpenAI engineers reported a software-based optimization that could cut inference costs by more than half for some models, first reported by The Information. Separately, on June 24, OpenAI and Broadcom unveiled the Jalapeño chip, a custom ASIC designed specifically for LLM inference, with initial deployment planned for end of 2026.
How do I reduce AI inference costs without changing the model?
The highest-ROI moves are: (1) model routing - send simple subtasks to smaller models and reserve frontier models for complex reasoning; (2) context compression - trim low-signal tokens before each call; (3) semantic caching - reuse responses for repeated queries. Together, these can reduce inference spend by 60-80% with minimal quality impact.
What is cost per task and why does it matter more than cost per token?
Cost per task is the total inference spend to complete one unit of useful work - a resolved ticket, a summarized thread, a drafted document. Cost per token measures only the input to a single model call. In agentic systems with multiple calls per task, cost per token can look low while cost per task is unsustainably high. Measure the outcome, not the unit.