It is 11 AM on a Tuesday and the on-call engineer is staring at a classification pipeline that keeps misreading edge cases. The prompt is already 600 tokens long. There's a chain-of-thought block, twelve curated examples, a rubric, a disclaimer. It still trips on anything that doesn't look exactly like the training examples someone wrote in Notion six months ago. Someone in the thread suggests fine-tuning. Someone else says the prompt just needs more work.
Both people are probably wrong about why.
Fine-tuning and prompting are not two ways to solve the same problem. They optimize different things through different feedback channels.
Fine-tuning updates the weights - the feedback is a gradient computed against a loss, whether that loss comes from token-prediction error, preference pairs, or a verifier. The model changes permanently.
Prompt engineering edits the input. The feedback can be anything: an eval score, a stack trace, a judge model, a human note. The model never changes; what changes is the conditioning.
That distinction sounds academic until you have a production system running a million classifications a month.
When the numbers make the case
A 2025 study by Highlighter.ai classified power outage reports and serious workplace injury reports, pitting a fine-tuned Qwen2.5-7B against Claude Sonnet 3.5 and 3.7 with prompt engineering. The fine-tuned 7B model hit 88% accuracy on power outages versus 31% for prompted Claude. On serious injury classification: 78% versus 59%.
That accuracy gap matters, but the cost gap is what ends the argument. At inference scale, the 7B model cost $789 per million classifications; prompted Claude cost $11,485 per million. The 14× cost gap came almost entirely from token efficiency - the prompted model needed an exhaustive instruction set on every call.
The prompt was doing work that weight updates should have been doing. Every call was re-explaining the task from scratch because the base model had no internalized knowledge of what "serious workplace injury" means in this specific regulatory context. Prompt optimization discovers what decomposition or strategy works for your task. Fine-tuning bakes that decomposition into the weights so you don't keep paying for it in tokens.
The honest case for staying with prompts
Here's the steelman: most teams asking about fine-tuning aren't ready to fine-tune. Most teams asking about fine-tuning should not fine-tune. They should fix their prompts, build a real RAG pipeline, and write evals - in that order.
Prompt engineering and few-shot examples solve 70% of LLM performance problems. Fine-tuning is the 30% solution. That's not a knock on prompting. It's a sequencing argument. If you haven't written evals yet, you cannot measure whether a fine-tuned checkpoint is actually better - you're just spending compute and hoping.
There's a second reason to stay cautious. Fine-tuning is for form, not facts. You use it to shape behavior, style, structured output, and refusal patterns - not to inject knowledge that changes weekly. A fine-tuned model that classifies incident reports beautifully will go stale the moment regulatory definitions shift, and updating it means another training run. A RAG pipeline or a prompt edit handles that change in an afternoon.
Fine-tuning a model on analytics dashboards to "speak BI language" may reduce its ability to reason about unstructured data or perform broader tasks that weren't part of the training set. Unless the task is stable and narrow, sticking to prompting or RAG avoids locking the model into a narrower skill set.
The right sequence
The right sequence in 2026 is: Prompt → RAG → Fine-tune → Distill. Most teams stall somewhere in the middle because they haven't written evaluations precise enough to tell whether they've exhausted the prior step.
Fine-tune when prompt engineering has hit a performance ceiling, you need consistent output format, you have 500+ high-quality examples, or latency and cost make large models impractical. Fewer than three of those things are true for most tasks most teams build.
For the tasks where all four are true - stable label sets, abundant examples, repetitive volume, fixed output schema - the case flips hard. If you have a narrow, repetitive task, a fine-tuned 8B parameter model can perform as well as a 70B model on that specific task. At millions of calls per month, that cost difference is significant.
The narrower and more repetitive the task, the more wrong it is to be paying frontier-model prices for it. An AI teammate like Beagle handles open-ended retrieval and reasoning across Slack and Teams - the kind of flexible, multi-topic work where a general model earns its keep. But if you're running a batch classifier on incident tickets, a fine-tuned small model running on your own infrastructure is probably the correct answer, not a bigger prompt to a bigger model.
The support ticket that broke the prompt on that Tuesday probably needed fine-tuning. But the team needed evals before they needed a training run. Those aren't the same bottleneck, and mixing them up is how you end up with a fine-tuned model that nobody can verify is better than what it replaced.