A fine-tuned Qwen2.5-7B model hit 88% accuracy on a power outage classification task; Claude Sonnet with prompt engineering hit 31%. At inference scale, the fine-tuned model cost $789 per million classifications versus $11,485 for the prompted Claude - a 14× gap that came almost entirely from token efficiency, because the prompted model needed an exhaustive instruction set on every single call.
That number is real, sourced, and worth sitting with. It is also incomplete. The task was fixed-label classification with abundant training examples. The interesting question is what counts as a "fixed label set with abundant examples" - and in 2026 that excludes most agent workflows, most coding tasks, most retrieval-grounded QA.
That gap between the headline number and the actual applicability is the whole decision. Teams that miss it either overbuy fine-tuning for work that would be fine with a better prompt, or underbuy it for narrow tasks where they are burning money on tokens every hour.
When fine-tuning an AI agent actually makes sense
Fine-tuning moves behavior into weights instead of instructions. Prompting changes what you tell the model at runtime - instructions, examples, structure - without touching the model's weights. Fine-tuning changes the model itself, training it on examples so behavior is baked in. Prompting is iterated in minutes and reverted instantly; fine-tuning takes a training run, a dataset and evaluation, and rolling it back means redeploying a previous model.
The structural advantage of fine-tuning is shorter prompts. A fine-tuned model does not need a 4,000-token prompt to know your tone, because the tone is now part of who it is. Prompts get shorter, outputs get more consistent, and you can often run a much smaller, cheaper, faster model at quality that rivals a frontier model on your narrow task.
Frontier LLMs in 2026 are excellent at general instruction following. They still fail at three things that fine-tuning fixes: exact output schema, narrow domain vocabulary, and brand voice. A base model can describe a SOC 2 control in plausible English, but it will not consistently emit the JSON your downstream parser expects, will not use your internal product nicknames, and will drift away from your support team's voice.
The bad candidates are equally specific: fine-tuning beats prompting when a task is narrow, high-volume, and stable. Poor candidates include broad multi-task agents, fast-changing requirements, and anything below roughly 50,000 requests per month on a single task.
The Prompt → RAG → Fine-tune → Distill sequence
This sequence matters because each step adds maintenance cost.
The right sequence in 2026 is Prompt → RAG → Fine-tune → Distill, and the highest-ROI fine-tuning is a thin LoRA or QLoRA adapter on top of a strong base model, paired with retrieval rather than replacing it. You do not jump to the right end because it sounds serious. You earn your way there when the cheaper option provably hits a wall.
Start with prompt optimization - DSPy plus GEPA - which now beats GRPO by 6 to 19 points using 35× fewer rollouts. Move to SFT only when you need a smaller model or a frozen behavior. Add RL (GRPO via Unsloth or ART) only when verifiable rewards exist and the prompt-optimized ceiling is too low.
Here is a practical read of which step fits which scenario:
| Situation | Right move | Why |
|---|---|---|
| Output format drifts despite clear instructions | Fine-tune (LoRA) | Schema baked into weights, not re-parsed each call |
| Task uses knowledge that changes weekly | RAG + prompting | Retrieval reads from source at question time |
| High-volume, fixed labels, abundant examples | Fine-tune small model | 14× cost gap at scale; shorter prompts reduce token spend |
| Multi-step agent, open-ended reasoning | Prompt optimization first | Task shape changes too often to freeze into weights |
| Need 70B quality at 7B inference cost | Distillation | Compress teacher outputs into student weights |
Form that is stable - meaning you have genuinely decided how the model should behave - is a candidate for fine-tuning. Form that is still in flux belongs in a prompt, because writing an unstable preference into model weights is an expensive way to lock in a decision you have not made yet.
Beagle in action#ops-data, 11:02amWhat happened to OpenAI fine-tuning, and what to use instead
There is a wrinkle in the tooling right now. As of mid-2026, OpenAI has wound down self-serve fine-tuning access; alternatives include Gemini, Mistral, and open-source LoRA adapters.
OpenAI began winding this down citing that better base models have closed most of the quality gap that made fine-tuning necessary.
That claim is partially true and partially convenient. Better base models do handle more tasks without fine-tuning. But they close the gap on general tasks - not on the classification-at-scale, schema-consistency, and domain-vocabulary cases where fine-tuning's cost advantage is widest.
Fine-tuning adds a one-time training cost and, on OpenAI models, a 50-100% inference premium per request. On Google Gemini and Mistral, there is no inference premium, making fine-tuning more attractive at lower volumes. The training compute spread is also wide: from $0.48 per million tokens for open-source 7B models on Together AI to $25 per million tokens for GPT-4o on OpenAI.
For teams that want to self-host, a 70B-class open-weight model fine-tuned on a specific task often matches a frontier API model at a fraction of the inference cost. The 2026 cost-per-quality sweet spot for many production agents is a QLoRA 8B-class or 70B-class open-weight model served on your own infrastructure, with RAG on top for fresh knowledge.
What this means for teams running agents in Slack and Teams
The relevant question is not "should we fine-tune" in the abstract. It is: which parts of the agent's job are stable enough that paying to bake them in makes sense?
A Slack-resident agent typically does a mix of things: lookup, triage, drafting, routing, and summarization. Most of those tasks have variable inputs and shifting requirements - which means prompting (or prompting plus RAG) is the right tool. The cases where fine-tuning earns its keep are narrower: consistent JSON emission for downstream integrations, a specific classification against a fixed taxonomy (ticket severity, department routing, compliance flags), or enforcing a voice that the base model consistently gets wrong.
The practical reason prompting comes first for agents specifically is that most of what an agent needs to get right changes too often to bake into weights. Business rules get updated. Policies change. A product catalog shifts weekly. Fine-tuning something that will need to be retrained in six weeks is not a cost saving - it is a maintenance liability that shows up on a future sprint.
A teammate like Beagle, living in Slack or Teams and drafting replies for human approval, benefits from this clarity too: the prompting and RAG layer handles context and freshness, while any narrow formatting or classification task that fires thousands of times a day is the actual candidate for a fine-tuned adapter.
Fine-tuning vs. prompting for AI agents: common questions
When does fine-tuning beat prompting for AI agents?
Fine-tuning wins on narrow, high-volume, stable tasks where a long system prompt would otherwise be sent on every request. Classification with a fixed label set, consistent JSON output, and domain vocabulary are the clearest cases. Below roughly 50,000 requests per month on a single task, the training and curation cost rarely breaks even.
Is fine-tuning still worth it now that frontier models are stronger?
Yes, in a narrow lane. Stronger base models close the gap on general tasks. They do not close the cost gap on high-volume classification, where a fine-tuned 7B model can run at 14× lower cost than a prompted frontier model. The question is volume and stability, not capability alone.
What is the right order for tuning an agent?
The canonical 2026 sequence is: Prompt, then RAG, then Fine-tune, then Distill. You do not jump to the right end because it sounds serious. You earn your way there when the cheaper option provably hits a wall.
Why did OpenAI wind down self-serve fine-tuning?
OpenAI began winding this down citing that better base models have closed most of the quality gap that made fine-tuning necessary. Alternatives for hosted fine-tuning include Gemini and Mistral, both of which charge no inference premium on fine-tuned models. Open-weight LoRA adapters via vLLM or Together AI are the self-hosted path.
Does fine-tuning replace RAG for keeping agents current?
No. RAG reads from the source at the moment of the question, so the next update is reflected the next time someone asks. A fine-tuned model that memorized last quarter's policy is stale the day the policy changes, and retraining is not a same-day fix the way editing a document or a prompt is. The two are complementary: fine-tune the interface; retrieve the content.