A team running a customer support pipeline through GPT-5 at 50 million tokens a month is paying frontier rates for pattern matching that a 13B-parameter model could handle. A 13B fine-tune that lands within two accuracy points of a frontier model on a locked eval set produces the highest-leverage cost reduction available on a stable workload - inference runs 12x to 40x cheaper per token. That is not a theoretical number. It is the actual arithmetic once distillation goes right.
The problem is that distillation goes wrong quietly, and most teams only notice after the support escalation rate has already climbed.
What distillation actually means here
Model distillation trains a smaller model - the student - using the predictions, features, or internal structure of a larger, pre-trained model - the teacher. You're teaching the student to replicate the teacher's reasoning, not just its answers. That distinction matters in practice: a student trained only on final outputs tends to overfit to confident examples and fail on the edge cases where the teacher showed uncertainty or took a multi-step path.
Distillation fine-tuning is most effective when the teacher model is substantially more capable than the student on the target task. It is recommended for transferring complex, multi-step reasoning capabilities, including scientific, medical, and domain-specific question answering that requires step-by-step reasoning - tasks where a teacher with chain-of-thought behavior consistently produces higher-quality responses than the student.
Distillation provides smaller gains on tasks where the student already performs close to the teacher, or on short-form retrieval tasks where the teacher's reasoning trace adds no value. If your task is a five-token classification, distillation is probably not the lever to pull. The case for it is strongest where the teacher's reasoning chain is genuinely what you want the student to inherit.
The three conditions that make the case hold
Distillation wins when the workload is narrow, the volume is above 50 million tokens per month, and the eval set is locked. Missing any one condition turns distillation into a regression risk.
"Locked eval set" is doing real work in that sentence. If your task definition is still shifting - new edge cases, changing output formats, evolving user intent - then you will retrain constantly and the economics collapse. Distillation is a bet that you understand the task well enough to freeze it. Teams that reach for it too early end up with a student model that was trained on yesterday's version of the problem.
The volume threshold matters because the engineering cost of a distillation project is real. The empirical 2026 cost band for a narrow-workload distillation project starts around $35,000 when you account for data curation, training compute, and the eval harness you need to trust the student in production. Below a certain token volume, you never recover that cost against the frontier API bill.
The case fails on open-ended workloads where the long tail is too sparse to distill, but it wins decisively on the narrow, high-volume task patterns that dominate enterprise AI spend.
The failure modes that show up after deployment
The optimistic framing of distillation leaves out the part where it competes with the production system it replaced. Developers feel the pain when a distilled model passes a benchmark but fails the product's odd cases. SREs see lower latency beside rising fallback rate, tool timeouts, and retry storms.
In multi-step agent pipelines, the risk compounds because the student may act as planner, retriever, tool caller, and final writer. A small loss in each step can become a failed workflow even when each individual response looks acceptable.
This is why the eval set has to be production-shaped, not benchmark-shaped. The measurable symptoms are: eval-fail-rate-by-cohort rising, JSONValidation failures clustering around one route, completion token count falling while support escalations rise, or p99 latency improving while task success drops. A model that looks better on every infrastructure metric can still be degrading the product.
One underappreciated failure mode is what happens when the teacher's refusal or hedging behavior gets stripped in distillation. Distillation reduces inference spend but increases the team's responsibility for failure modes the frontier vendor would otherwise own. Self-hosted distilled models need explicit coverage for cases the frontier model handled by default - jailbreaks, PII mishandling, confidently wrong outputs.
When to pick distillation over a smaller base model
There is a version of this question that gets asked wrong: "should we distill, or just use Qwen 3 8B directly?" The answer depends on whether the teacher's reasoning traces add information the student could not learn from the task data alone.
A 70B model fine-tuned entirely on a specific corporate codebase can beat a frontier model's generalized coding capabilities within that environment. But that is fine-tuning on proprietary data, not classical distillation. Classical distillation - using the teacher's soft outputs and intermediate reasoning - earns its cost when the task has a reasoning structure the teacher has already internalized and the student cannot infer from labeled examples alone.
Distilling reasoning behavior into a smaller model allows you to bake the reasoning directly into the weights, bypassing the need for long multi-step inference-time compute loops. That also means the student is faster at inference time for tasks that would otherwise require chain-of-thought prompting on a smaller base model - a real operational win in latency-sensitive pipelines.
The honest summary is that distillation is a mature technique with a real production track record, but it has a narrow sweet spot. If your product needs broad, flexible intelligence - brainstorming, open-ended Q&A, multi-step planning - frontier models are hard to beat. The moment your workload becomes narrow and high-volume - classifying records, generating structured API responses, parsing contracts - fine-tuning and distillation start to make sense. The work of distillation is less in the training run than in the discipline required to lock the task definition before you start.
A teammate like Beagle, which handles structured Slack and Teams workflows, sits in exactly the category where distillation eventually makes sense: the task is narrow, the volume is predictable, and the outputs can be evaled against a known schema. That is not an accident. It is the shape of any AI feature that gets used reliably at scale.