Fine-Tuning Beats Prompting on One Thing. Maintenance Isn't It.

Training a fine-tuned GPT-4.1-mini SFT job costs around $5-$50. Deployed inference on that tuned model runs roughly 2× the base model price. Teams read those numbers and think fine-tuning is cheap. They are looking at the wrong line item.

The actual cost of fine-tuning is not the training run. It is everything that comes after: the eval harness you have to maintain, the retraining cadence every time your task distribution shifts, and the engineering hours you spend re-running the whole pipeline when the base model updates underneath you. The least-discussed cost of fine-tuning is the long-term maintenance burden. It is real and not negligible. When you ship a fine-tuned model, you inherit a commitment that lasts as long as the feature does: base model updates, data drift, task evolution, and infrastructure maintenance.

That is the debate nobody is having. The internet is full of "fine-tuning vs. prompting" comparisons that treat the question as purely technical - does the fine-tuned model score higher? It usually does. But scoring higher on your eval is not the same thing as being the right architectural choice.

Where fine-tuning actually earns its keep

Fine-tuning wins on a specific, narrow set of problems. There are four places where fine-tuning genuinely moves the needle today: structured output reliability when prompt-only solutions still hallucinate fields, domain vocabulary or jargon that base models hedge on, refusal and tone control where prompt instructions get overridden, and cost compression through small-model distillation from a working large-model pipeline.

The quantitative case is real. A good prompt can yield reasonable results, but fine-tuning improves quality by 10% on average for domain-specific tasks requiring structured outputs - this was measured against a full suite of closed and open-source LLMs, not a toy benchmark. On a task like generating low-code workflows in JSON, a fine-tuned 12B model beat every prompted LLM in the comparison set, including GPT-4o.

That 10% matters if you are running millions of calls and quality is your bottleneck. It is less decisive if you are running 5,000 calls a month and your actual problem is that a domain-expert reviewer is still catching edge cases a better system prompt might have handled.

Use fine-tuning when you need strict output formats (JSON or tool calls), a consistent tone, or strong performance on a narrow task. The guidance is sound. The part that gets dropped from most writeups is the prerequisite: you need a stable task with a stable schema and labeled data that does not go stale. Baking facts into weights produces stale, unverifiable answers and erodes the model's general capability through catastrophic forgetting. If your problem is "the model doesn't know our docs," fine-tuning is the wrong tool.

The maintenance math most teams skip

Here is where the framing breaks. Teams compare the training cost against the token savings at inference and conclude fine-tuning is a good deal. That comparison omits the denominator.

Retraining cadence for active products is typically every 1-3 months. Engineering time runs roughly 5-15% of one engineer's capacity per fine-tuned model. Re-evaluation overhead on every base model update: 1-2 days per cycle. Multiply that across even a small portfolio of task-specific models and you have a real operational commitment.

Fine-tuned models are also version-locked. When the base model updates, your fine-tune breaks. Fine-tunes are version-locked. There is no migration path. The only option is to retrain from scratch - reprocessing your data, reformatting your JSONL, rerunning your jobs, and repaying for all of it. This is not a theoretical risk; base models are periodically updated to incorporate new knowledge, improve safety, or enhance general capabilities, while downstream model services built on previous versions must be maintained accordingly. This creates a service lifecycle challenge: updating the base model may invalidate existing task adapters, and retraining numerous adapters from scratch can introduce substantial computation cost and rollout latency.

A team running five fine-tuned models in production should expect roughly 0.5-1 FTE of ongoing maintenance load, or $80,000-$200,000 per year in fully loaded engineering cost. The training run itself - even OpenAI's RFT at $100 per training hour - is a rounding error against that number.

Steelmanning fine-tuning: where the critics are wrong

The case against fine-tuning is often overstated. Here is where critics miss.

Fine-tuning makes financial sense in specific scenarios: replacing an expensive model with a cheaper fine-tuned one, or eliminating large system prompts that add 2,000+ tokens per call. If you have a use case making millions of calls per month with a 3,000-token system prompt, a fine-tuned small model can reduce that to near zero - the cost savings are not marginal, they are orders of magnitude.

OpenAI's Reinforcement Fine-Tuning (RFT) also opens a genuinely new category. In RFT, the reward signal comes from a custom grader that you define for your task. For every prompt in your dataset, the platform samples multiple candidate answers, runs your grader to score them, and applies a policy-gradient update that nudges the model toward answers with higher scores. This cycle - sample, grade, update - continues across the dataset until the model reliably optimizes for your grader's understanding of quality. For tasks where correctness is verifiable - code execution, structured extraction with a schema validator, mathematical reasoning - RFT sidesteps the hardest part of SFT, which is writing enough good labeled examples to cover the distribution.

Supervised fine-tuning costs literally pennies in comparison to RFT, but RFT does not need labeled outputs. It needs a grader. For teams who can write a deterministic grader - a SQL validator, a unit test runner, a JSON schema check - that is a meaningful shift in what fine-tuning requires.

The honest version of the debate is a table:

The decision the framework keeps hiding

Most teams reach for fine-tuning because they hit a wall with prompting and want to feel like they are doing something structural. That impulse is understandable. But most teams asking about fine-tuning should not fine-tune. They should fix their prompts, build a real RAG pipeline, and write evals - in that order.

Hidden costs include data preparation - 10-40 hours of labeling - evaluation infrastructure, and the 5-15 experiment iterations most teams need. Those hours do not show up in the OpenAI billing dashboard, but they are the real budget line.

The right sequence, borrowed from practitioners who have run this at scale: fix the prompt first, add retrieval if the problem is missing knowledge, build evals no matter what, and reach for fine-tuning only when a specific behavioral gap survives all three. Fine-tuning is for form, not facts. You use it to shape behavior, style, structured output, and refusal patterns - not to inject knowledge that changes weekly.

A teammate like Beagle - operating inside your team's Slack channels - is doing exactly this every time it answers a question from a doc or thread: retrieval plus a strong base model, no weights modified, no retrain scheduled when the doc changes. That is not a limitation. It is the architecture choice that ages well.

The question is not whether fine-tuning makes the model better at a task. It usually does, by about 10%. The question is whether your team can operate what you are building for the next eighteen months.

Fine-tuning vs. prompting: common questions

When does fine-tuning actually beat prompting?

Fine-tuning beats prompting when you need strict, stable structured outputs - JSON schemas, consistent tone at scale, or domain vocabulary a base model hedges on - and your task distribution is unlikely to change weekly. Expect roughly a 10% quality improvement on structured tasks. It does not beat prompting for knowledge retrieval; use RAG for that.

How much does fine-tuning actually cost end-to-end?

The training run is cheap: $5-$50 for a GPT-4.1-mini SFT job, or $100/hr for OpenAI's RFT on o4-mini. The real cost is maintenance - data prep (10-40 engineering hours), eval infrastructure, and retraining every 1-3 months. A small team running five models should budget $80,000-$200,000 per year in fully loaded engineering time.

What happens to a fine-tuned model when the base model updates?

Fine-tuned models are version-locked to the base model they trained on. When the provider updates the base - new safety tuning, architecture change, tokenizer update - the adapter does not migrate automatically. You retrain from scratch: reprocess data, reformat JSONL, rerun evals, repay training costs. Plan for this in your maintenance calendar.

Is OpenAI's Reinforcement Fine-Tuning (RFT) worth the cost premium?

RFT at $100/hr is 100-700× more expensive per training hour than SFT, but it removes the need for labeled output examples. If you can write a deterministic grader - a schema validator, a unit test runner, a domain-specific scorer - RFT can outperform SFT on verifiable tasks. If you cannot write a reliable grader, the cost premium is hard to justify.

What should I fix before even considering fine-tuning?

In order: write a stronger system prompt with concrete examples, add a RAG layer if the failure mode is missing or stale knowledge, build an eval set of at least 100 representative examples, and measure whether the failure rate is actually unacceptable. Most teams who do this discover the eval set reveals a prompt fix, not a training run.