OpenAI stopped reporting SWE-bench Verified scores in February 2026. Not because their model performed badly. Their Frontier Evals team announced they would stop reporting SWE-bench Verified-the 500-issue benchmark that had become the unofficial scoreboard for "can the machine code yet"-because the score stopped meaning anything.
That single decision should change how every engineering lead reads a model leaderboard. The headline number you used to compare open-weight models is now partly measuring memory, not skill.
Why the benchmark leaderboard is not a ranking
As of mid-2026, the top of the SWE-bench Verified board is a cluster of self-reported 80%-to-95% scores measured on vendor scaffolds against a contamination-prone public set-useful as a tier filter, misleading as a ranking.
The two problems compound each other. First, contamination: the benchmark's 500 GitHub issues have been public long enough for solutions to seep into training corpora. Cracks had been forming for months, and OpenAI's decision to abandon Verified scores in February 2026 brought the debate into the open. The core problem is twofold: the fixed dataset has been public long enough for contamination to seep into training data.
Second, harness dependency: the agent scaffold around a model can move a score by 10 to 20 points without changing the model at all.
Every entry on the board is vendor self-reported; none is independently verified, and harness differences apply. When two vendors report different numbers for the same base model, the harness is usually the explanation-not capability.
Ranks 6 through 10 on the June 2026 board span 0.4 points (80.6% to 80.2%), with two open-weights models inside that band. When five models from four labs sit statistically tied near 80%, the benchmark has stopped discriminating at the frontier.
The same decay pattern holds for older knowledge benchmarks. The most-cited example is HumanEval: frontier models report near-saturated scores on its 164 problems, which have been thoroughly discussed in pre-training data, while the same models score visibly lower on the contamination-resistant LiveCodeBench. The practical fix: triangulate three benchmark types before you trust a ranking-read a static academic eval (MMLU-Pro, GPQA), a human-preference arena (LMArena), and an agentic suite (SWE-bench, Terminal-Bench). Agreement across all three is the signal; a single leaderboard number is close to meaningless on its own.
Where the open-weight frontier actually sits
Strip away the benchmark noise and the genuine picture is striking. The best open-weight models are within 3-6 points of the leading proprietary models on the Artificial Analysis Intelligence Index. For context: just one year ago the highest-scoring open-weight model was DeepSeek V3 0324, which achieved 22 on the Intelligence Index-13 points below Claude 3.7 Sonnet at 35. The gap closed by roughly half in twelve months.
The current field, honestly assessed:
| Model | Architecture | Key strength | License | Self-host reality |
|---|---|---|---|---|
| GLM-5.2 (Z.ai) | 744B MoE, 40B active | Reasoning, long-context coding | MIT | Needs ~245 GB combined memory at 2-bit quant |
| DeepSeek V4 Flash | 284B total, 13B active | Price-performance for agents | MIT | Multi-GPU; "13B active" ≠ small footprint |
| Kimi K2.7 Code | 1T total, 32B active | Coding agents, vision | Modified MIT | Data-center scale only |
| Qwen3.6-35B-A3B | 35B total, 3B active | Single-GPU deployment | Apache 2.0 | Fits one consumer GPU; first open model to pass 70% SWE-bench at that scale |
| Nemotron 3 Ultra | 550B / 55B-active MoE | US-built, fully open provenance | OpenMDW | 4-8× H100 required |
As of July 2026, GLM-5.2 is the strongest all-round open-weight LLM in most comparisons, while Kimi K2.7 Code stands out for coding agents and Gemma 4 12B is a practical laptop model. The right choice still depends on workload, hardware, and licence.
One non-obvious signal worth noting: the calculus shifted in April 2026 with the arrival of Qwen 3.6-35B-A3B. The 3B-active MoE architecture runs at interactive latency on a single consumer GPU, and the Apache 2.0 licence removes all commercial-use concerns. For organisations processing millions of tokens daily on coding or retrieval workloads, this is the first open-weight model where the self-hosting payback period is measured in days rather than months.
Meanwhile, on the closed side, GPT-5.6 comes in three variants-Sol, Terra, and Luna-and OpenAI claims Sol is 54% more token efficient on AI coding tasks. That efficiency claim matters because it shifts the cost comparison: a model that produces the same result in half the tokens is cheaper per task, not per token.
The self-hosting math most teams get wrong
The model choice and the deployment choice are separate decisions, and teams routinely collapse them. Self-hosting open-weight LLMs is the deployment decision teams get wrong most often in 2026-not because the engineering is hard, but because the cost intuition is backwards. Owning the GPU feels cheaper than paying a per-token API bill, yet for most workloads the math never crosses over.
Against frontier closed-source models like GPT-4.1 or Claude 4, self-hosting on a reserved cloud GPU breaks even at roughly 2M to 5M tokens per day compared to frontier API pricing. That sounds achievable. But the GPU invoice is not the whole bill.
At roughly 10% utilisation, your real cost per token can be about 10× the headline GPU rate-enough to make an idle H100 more expensive per token than a premium frontier API. Self-hosting economics only work with high, sustained throughput. A team running a coding assistant that fires 200 requests a day is almost certainly not there.
There is a third path that most comparison posts skip: open-weight hosted APIs from providers like Together.ai and Fireworks occupy a compelling middle ground that undercuts both proprietary APIs and self-hosting at light and medium tiers. You get the open-weight price without the ops overhead. The break-even against those providers is much harder to hit- often requiring 50M+ tokens per day-because they already operate optimised infrastructure at scale.
Compliance changes the entire equation. For HIPAA, GDPR, or SOC2-bound workloads, the breakeven analysis is often moot. If your data cannot leave your infrastructure, self-hosting is not a cost question; it is a legal one. That is the clearest case for doing it regardless of the spreadsheet.
One infrastructure note that is easy to miss: Hugging Face moved Text Generation Inference to maintenance mode in December 2025. New greenfield deployments should default to vLLM for batch throughput and SGLang for prefix-heavy RAG and multi-turn workloads. Teams still running TGI in production have a live technical debt item.
The licensing gap nobody talks about
The open-weight label on a model card does not mean what most people assume. The term "open source" means something different in the LLM world than in traditional software. A fully open model publishes weights, training data, architecture code, and training pipeline under a licence allowing free use, modification, and redistribution. In reality, most models called "open source" are really "open weight": only the model weights are publicly available, but the training data and pipeline remain proprietary.
Apache 2.0 licensing terms mean a downstream organisation can download, fine-tune, and redistribute the model commercially without seeking the vendor's permission or triggering a legal review-no custom licence terms, no usage caps based on user scale. MIT is similarly permissive. Many other licences include commercial-use restrictions or scale thresholds buried in appendices.
Beagle in action#eng-tools, Tuesday afternoonOpen-weight model benchmarks: common questions
What does SWE-bench Verified actually measure?
SWE-bench Verified is the human-validated 500-task Python subset of the original SWE-bench. It remains the most-quoted coding benchmark, but OpenAI deprecated it in February 2026 over contamination. It still separates weak models from strong ones-it just cannot rank the top dozen reliably. For production model selection, use SWE-bench Pro's scores on private repos.
Why do two vendors report different scores for the same base model?
The gap between Scale's standardised harness and a vendor's tuned agent harness is typically 10-30 points, and it is mostly context retrieval and tool-use quality, not model capability. A vendor's best scaffold bakes into their headline score. When comparing models across labs, check that the evaluation harnesses are comparable before drawing conclusions from a rank difference inside 5 points.
When does self-hosting an open-weight model actually save money?
Self-hosting saves money at sustained high volume with high GPU utilisation. For most enterprise workloads in 2026, a managed API is cheaper and faster to ship than self-hosting until you reach very high, sustained token volume. Self-hosting an open-weight LLM carries a largely fixed monthly cost, so it only wins on price at scale or under strict data-residency rules. The compliance override-HIPAA, GDPR, SOC2-can make self-hosting mandatory regardless of cost.
Is the open-weight quality gap to closed models still large?
No, and the speed of change is the more important signal. In April 2026, open-weight models routinely match or exceed the performance of proprietary models from twelve months ago, and in specialised benchmarks they compete with the very best closed models available today. The remaining gaps are in instruction-following polish, multimodal capability, and very long contexts-not raw benchmark scores on standard tasks.
How should I read a model card score before picking a model?
A provider may report SWE-bench with a custom coding agent, while another uses a minimal shell loop. Reasoning models may also receive different token budgets. Before trusting any single score: note who ran the eval (vendor vs. independent), check whether it's on the contaminated public set or a private set, cross-reference against at least one human-preference arena result, and verify the licence terms match your commercial use case.