A paper accepted as an oral at ICLR 2026 showed a prompt optimizer called GEPA beating GRPO - a popular reinforcement learning fine-tuning method - on six benchmarks, while using up to 35 times fewer rollouts. If you have been treating "fine-tune or prompt" as a binary choice, that result should give you pause, because you may be skipping the option that wins.
This is not an abstract research finding.
GEPA is integrated into DSPy as dspy.GEPA
, so any team already using DSPy can run it today. And the underlying research shift it represents - prompt optimization as a real engineering discipline, not just hand-tuning a system message - is one of the more practical things to come out of the last few months of AI work.
Why the fine-tune vs prompt question is now the wrong frame
A year ago the trade was simple: prompts were cheap and brittle, fine-tuning was expensive and powerful. In 2026 both halves of that sentence are wrong. Prompt optimization has become a real engineering discipline that beats reinforcement learning on its own benchmarks. Fine-tuning has become a commodity you can run on modest hardware. The boundary between the two has dissolved.
The practical consequence: a team that reaches for GRPO or LoRA before exhausting structured prompt optimization is spending compute it does not have to. Start with prompt optimization (DSPy + GEPA), which now beats GRPO by 6 to 19 points using 35x fewer rollouts. Move to SFT only when you need a smaller model or a frozen behavior.
That said, fine-tuning does have a specific, well-defined job in 2026. Fine-tuning is for form, not facts. You use it to shape behavior, style, structured output, and refusal patterns - not to inject knowledge that changes weekly. The teams who get this wrong are the ones asking fine-tuning to do RAG's job: they want the model to "know" their product catalog, so they train on it. Two weeks later the catalog changes and the model is stale.
What GEPA actually does, and why it matters for teams
GEPA argues that the interpretable nature of language provides a much richer learning medium for LLMs compared to policy gradients derived from sparse, scalar rewards. It samples trajectories - reasoning steps, tool calls, outputs - and reflects on them in natural language to diagnose problems, propose prompt updates, and combine complementary lessons.
The key intuition: a scalar reward of 0.43 tells the model less than a sentence describing what failed. Instead of running thousands of rollouts to nudge model weights, GEPA runs far fewer, reads the failures in natural language, and edits the prompt directly. To avoid local optima, GEPA maintains a Pareto front: instead of evolving only the global best prompt, it stochastically explores the top-performing prompts for each problem instance. This diversification enables robust generalization and mitigates getting stuck in local minima.
For a product team, the implication is concrete: before you start labeling hundreds of examples for fine-tuning, you can run GEPA against a scoring function you already have - your existing eval suite, your test cases, your rubric - and let it find a better prompt automatically. A teammate like Beagle, running in the same Slack channel where your team is already iterating on prompts, can surface the before/after diff for a human to review before it goes anywhere.
Beagle in action#ml-infra, 11:02amWhen fine-tuning still wins
None of this makes fine-tuning obsolete. There is a narrow, real set of cases where it is the right tool.
You need strict output formatting. Fine-tune a model to always return JSON in a specific schema, XML with specific tags, or structured tables. It is possible with prompting, but fine-tuning gives you 95%+ reliability instead of 85%.
The real business case for fine-tuning that does not get discussed enough: a fine-tuned 7-billion-parameter model can outperform a 70-billion-parameter model on a specific, narrow task. If you are running inference at volume and paying per token, that gap is real money.
RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants. SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution. The practical read: use SFT when you need the model to do exactly what it was shown, reliably. Use RL (GRPO, RLAIF, or similar) when you need it to handle variations you did not train on. Use GEPA when you want that RL-style generalization without paying the compute cost of RL.
SFT is also necessary to initialize models for RL. Without SFT, RL struggles because the base model lacks basic instruction-following skills. So the sequence is not "skip SFT and jump to RL." It is: use the lightest-weight intervention that achieves the task floor, then layer.
There is also a real ongoing cost that most decision-write-ups skip. The operational tax is real. Adapter versioning, rollback plans, retraining cadence, and base-model drift management are recurring costs, not one-time work. When a hosted provider updates their base model, your adapter may degrade silently. Plan quarterly revalidation.
The decision sequence that holds up in practice
The right sequence in 2026 is prompt → RAG → fine-tune → distill. The highest-ROI fine-tuning is a thin LoRA or QLoRA adapter on top of a strong base model, paired with retrieval rather than replacing it.
Laid out as a practical gate:
- Does better context fix it? Add examples, add retrieved docs, restructure the prompt. Most problems end here.
- Does structured prompt optimization fix it? Run GEPA or DSPy's compiler against your eval suite. This is where the ICLR 2026 finding bites most teams - they skip this gate.
- Do you need a smaller or deployable model? SFT with LoRA or QLoRA. Behavior gets baked in; you lose flexibility, you gain speed and cost.
- Do you need generalization to unseen variants? RL (GRPO via Unsloth or ART) only if you have verifiable rewards and the prompt-optimized ceiling is too low.
- Do you need to shrink the whole model? Distillation - train a smaller model on the larger model's outputs, not the original task data.
For AI engineering leaders, the fine-tuning decision in 2026 is no longer "can we afford to do this" - parameter-efficient methods have made it accessible to teams without massive compute budgets. The decision is "does our use case actually need it," and answering that correctly determines whether fine-tuning delivers the reliability improvement it promises or simply adds engineering overhead to a problem prompting could have solved.
Fine-tuning vs prompting LLMs: common questions
What is the difference between fine-tuning and prompt optimization?
Prompt optimization (tools like GEPA, DSPy) improves model behavior by automatically evolving the system prompt against a scoring function - no weight changes, no labeled dataset required. Fine-tuning updates the model's weights on your examples. Prompt optimization is faster, cheaper, and more reversible; fine-tuning locks in behavior and enables smaller, faster models.
When should I fine-tune instead of prompting?
Fine-tune when you need output format reliability above ~90% at scale, when you need a small model to perform like a large one on a specific task, or when your domain vocabulary is private enough that it is not in any public training data. Everywhere else, exhaust prompt optimization first.
Does GEPA actually beat fine-tuning in practice?
Across six tasks, GEPA outperforms GRPO by 6 percentage points on average and by up to 19pp, while using up to 35x fewer rollouts. GEPA also outperforms the leading prompt optimizer, MIPROv2, by over 10 percentage points (e.g., +12pp on AIME-2025). These are ICLR 2026 Oral results on Qwen3 8B and GPT-4.1 Mini. The caveat: GEPA requires a scoring function. If you cannot score your outputs automatically, you cannot run it.
What does "SFT memorizes, RL generalizes" mean for a product team?
SFT is a memorizer, while RL is a generalizer. Models trained with SFT excel at tasks they have seen before but falter when faced with new, unseen variations. For product teams: SFT is the right tool when your edge cases are enumerable and stable. RL is the right tool when you need the model to adapt to variations you cannot anticipate. Most narrow task deployments - classification, extraction, structured output - have enumerable edge cases. SFT works there.
What is the real cost of fine-tuning that teams underestimate?
The real cost decision is not training compute - it is eval, data curation, and lifecycle ownership. Budget 3-5x the training cost for ongoing lifecycle ownership over the next 12 months. Every time the underlying base model updates, your adapter needs revalidation. That is a recurring engineering tax that prompt-based approaches do not carry.