On February 23, 2026, OpenAI's Frontier Evals team quietly announced it was stopping SWE-bench Verified reporting. Not because the score was bad. Because the score stopped meaning anything. That decision - one line in a blog post - changes how every engineering team should read any coding leaderboard from this point on.
This matters right now because the open-weight side of those leaderboards just had its most impressive month in memory. Kimi K3, Moonshot AI's 2.8-trillion-parameter MoE, launched July 16 with a 1M-token context; weights shipped July 27 under Modified MIT.
On the LLM-Stats composite as of early August, the best open-weight model - Kimi K3 - scores 55.4 against 57.2 for GPT-5.6 Sol and 56.5 for Claude Opus 5.
That is a difference of 1.8 points between the best model you can download and the best model you can only rent.
But composite scores flatten things. The per-benchmark picture is more complicated, and the benchmark that generated the most K3 hype is the one you should trust least.
What SWE-bench Verified was actually measuring by June 2026
SWE-bench Verified was designed to test whether a model can close real GitHub issues on real Python repositories - a genuinely useful thing to measure. Since its August 2024 release, the industry widely used it to measure progress on autonomous software engineering, and it became a standard metric in frontier model launches.
Then it ran into two compounding problems.
First, contamination. Researchers found that 32.67% of successful patches involved solution leakage, and models recalled file paths from training data up to 76% of the time for in-distribution files versus up to 53% for external ones.
Most developers do not track or disclose data contamination, which can lead to inflated performance scores that reflect memorization rather than true ability.
Second, the test oracle problem. OpenAI audited 138 of the hardest tasks - the ones their model kept failing - and found that 59% had material flaws: tests too narrow to accept a correct answer, or too loose to reject a wrong one. A model that patches around a broken test passes. That's not coding skill.
The result: as of June 2026, only 1 of the 100 results on the llm-stats SWE-bench Verified leaderboard was independently verified; the other 99 were submitted by vendors themselves.
Any Verified score above 80% "warrants scrutiny about harness and tool access" - which is precisely the band the entire top of the June 2026 board now occupies. When the leading dozen models cluster between 80% and 95% on a self-reported benchmark, rank order stops carrying signal.
The replacement benchmark, SWE-bench Pro, is more defensible. It has 1,865 total tasks - 731 public, 858 held-out, 276 commercial - spanning 41 repositories in Python, Go, TypeScript, and JavaScript, with tasks averaging 107.4 lines changed across 4.1 files. That's a materially harder test, and the held-out commercial set is harder to contaminate than public GitHub issues.
Nearly all major LLMs exhibit significant data contamination across popular benchmarks, and this effect grows with scale - larger models have a higher capacity for verbatim memorization. The biggest models, which dominate leaderboards, are also the most likely to have memorized parts of the tests they score highest on.
Kimi K3's numbers: which ones to trust
Kimi K3 posted a lot of numbers at launch. Here is a brief table of the ones worth examining versus the ones to set aside:
| Benchmark | Score | Contamination defense | Verified independently? |
|---|---|---|---|
| Terminal-Bench 2.1 | 88.3% | Moderate - continuously updated | Partially (community runs) |
| FrontierSWE | 81.2% | Unpublished methodology | No - Moonshot self-reported |
| SWE-bench Pro | - | Strong (held-out commercial set) | Not yet reported for K3 |
| GPQA Diamond | 93.5% | Good (expert-curated, not saturated) | Partially |
| Artificial Analysis Intelligence Index | 60 (tied with GLM-5.3) | Independent | Yes |
Moonshot reports 81.2 on FrontierSWE and 88.3 on Terminal-Bench 2.0; Artificial Analysis independently scores it 60, tied with GLM-5.3 for the open-lineage lead. The gap between a self-reported 81 and an independently measured 60 is not a scandal - they are different scales and different tasks - but it illustrates why you should always note the source.
The most interesting third-party data for K3 came from a real workload test rather than a standardized benchmark. Fireworks AI ran K3 against Claude Fable 5 on roughly 1,000 agentic tasks. The result was not winner-takes-all: K3 outperformed on security, crypto, and long terminal loops; Fable 5 won on multilingual and web/data visualization tasks. Neither model dominated. That kind of per-task breakdown is more useful for deployment decisions than any composite rank.
Humanity's Last Exam and SWE-bench Verified are weighted heavily by serious evaluators because: HLE was built to be hard to saturate, and SWE-bench Verified was human-filtered specifically to remove unsolvable or gameable tasks
- even if that filtering has since proven incomplete. The practical rule, per one team tracking the field: use composites to decide which three models to try, and never to decide which one to ship.
Beagle in action#eng-infra, 10:41amThe part the leaderboard does not show: open ≠ runnable
Kimi K3's weights are genuinely public. Moonshot published the full 2.8-trillion-parameter weights on July 27, and the Hugging Face repo holds 96 shards, the Kimi K3 License, and the technical report. This is a real open release, not a promise.
But open is not the same as usable on your infrastructure. The mixture-of-experts design keeps every expert resident in memory, so the weights alone need roughly 1.4 TB of VRAM at native 4-bit quantization - in practice, an 8x NVIDIA B200 node at minimum and 16x H200 for the full 1M-token context.
Moonshot recommends deploying K3 on large GPU clusters, with production deployments targeting supernode configurations of 64 or more accelerators.
The cost math is concrete. At a mid-market rate near $5.50 per GPU-hour, an 8x B200 node runs about $32,000 a month, so the break-even against the $15/M API sits above 2 billion output tokens a month. Below that volume, the managed API wins on both price and operational effort.
For comparison: DeepSeek V4-Flash and Mistral Small 4 bring near-frontier quality to 2-GPU setups. If your team needs a self-hosted open-weight model that fits on hardware you can actually own, K3 is not it. The honest split for most teams: use the K3 API for frontier-class tasks, and self-host a smaller open model - GLM-5.3 Flash, DeepSeek V4-Flash - for the high-volume, latency-sensitive work where owning the stack actually pays off.
The OSI Open Source AI Definition requires three components: model parameters, complete training and inference source code, and training-data information detailed enough to rebuild a substantially equivalent system. Releasing only the weights does not meet that bar. By the strict OSI definition, almost no leading open model qualifies as open source. That is worth knowing before your legal team reviews a deployment agreement.
How to actually read a leaderboard in August 2026
Use the live table for capability order, then check evidence status, license terms, memory requirements, and serving cost before choosing a deployment. That's the right sequence, in that order.
A few concrete heuristics that hold up in the current field:
Check the source column. Some leaderboard data comes from model providers; some from independent evaluation runs. The distinction matters enormously when scores are self-reported.
Prefer execution-based benchmarks for coding. The serious benchmarks use execution-based verification: tau-bench checks database state, SWE-bench runs the test suite, tau2-bench's pass^k measures whether the agent succeeds reliably across attempts.
Watch for harness effects. K3's coding tests used Moonshot's Kimi Code harness; harness choices can materially change results. The same model on a different scaffold posts a different number.
LiveCodeBench for contamination resistance. It pulls problems from competitive programming sites, each stamped with a publication date, and lets you score a model only on problems released after its training cutoff. Saturation gets harder when the test set refreshes every few weeks.
Run a small task-specific eval before you commit. No single benchmark captures real-world coding performance accurately - you need a combination of evals to get a meaningful picture. Twenty minutes building a held-out test on your own codebase beats an hour reading leaderboard commentary.
Coding order differs from overall order, and a single HumanEval or LiveCodeBench result should not be treated as a complete coding verdict. If your use case is code review or agentic coding, look at SWE-bench Pro and Terminal-Bench specifically - not the composite rank.
Open-weight model benchmark: common questions
What does the open-weight vs closed model benchmark gap look like in 2026?
On the LLM-Stats composite in early August 2026, Kimi K3 scores 55.4 versus 57.2 for GPT-5.6 Sol and 56.5 for Claude Opus 5. K3 ranks fifth overall - ahead of GPT-5.6 Terra, Claude Opus 4.8, and Meta's Muse Spark 1.1. That is 1.8 points between the best downloadable model and the best API-only one.
Why did OpenAI stop reporting SWE-bench Verified?
On February 23, 2026, OpenAI's Frontier Evals team announced it would stop reporting SWE-bench Verified - the 500-issue benchmark that had been the unofficial scoreboard for coding capability - not because the score was embarrassing, but because it stopped meaning anything. The primary causes were task contamination and a high rate of flawed test oracles in the problem set.
Can you self-host Kimi K3 on a standard GPU server?
No. The weights alone need roughly 1.4 TB of VRAM at native 4-bit quantization, which means an 8x NVIDIA B200 node at minimum.
Moonshot recommends production deployments on supernode configurations of 64 or more accelerators. For most teams, the API at $3/$15 per million tokens is cheaper below 2 billion output tokens per month.
Which open-weight models are genuinely runnable on two GPUs?
DeepSeek V4-Flash and Mistral Small 4 bring near-frontier quality to 2-GPU setups.
For a single consumer GPU, Gemma 3 27B at 16 GB VRAM and Phi-4 14B at 8 GB VRAM are the easiest to self-host. The right model depends on whether you're optimizing for raw capability or for the hardware you actually own.
How do I know if a benchmark score is independently verified?
Check the leaderboard's evidence column or methodology notes. As of June 2026, only 1 of the 100 results on the SWE-bench Verified leaderboard was independently verified; the other 99 were submitted by the vendors themselves. Independent sources like Artificial Analysis, vals.ai, and Kilo's Terminal Bench runs are worth more weight than provider technical reports.