One year ago, the highest score on OSWorld hovered around 20%. By December 2025, an open agentic framework called Agent S3 had crossed the human baseline. That's a real number worth sitting with. But a separate paper published the same year found that a top-scoring agent needed 12 minutes to change paragraph line spacing in a document - a task a person with basic computer experience finishes in under 30 seconds. Both things are true at once, and the gap between them is the whole story.
What the OSWorld computer use agent benchmark actually measures
OSWorld is a benchmark of 369 real computer tasks run inside live Ubuntu, Windows, and macOS environments. The executable environment allows free-form keyboard and mouse control of real applications, including OS, LibreOffice Calc, Impress, Writer, Chrome, VLC, Thunderbird, VS Code, GIMP, and multi-app workflows. A score is either pass or fail - the agent either completed the task in a real environment or it didn't. That's stricter than most NLP benchmarks, which makes it a reasonable signal for progress.
Agent S is an open source computer use agent framework from Simular. It takes a natural-language task, looks at the screen, and completes the task by clicking, typing, and scrolling in ordinary desktop and web applications - no API integration and no per-app scripting required. The architecture is model-agnostic: it runs on macOS, Windows, and Linux, and works with models from OpenAI, Anthropic, and open-weight providers.
The generational progression is worth tracking because it shows the pace of the field, not just the headline:
| Version | OSWorld score | When |
|---|---|---|
| Agent S (original) | 20.6% | October 2024 |
| Agent S2 | 48.8% | 2025 |
| Agent S3 (single run) | 62.6% | October 2025 |
| Agent S3 + Behavior Best-of-N | 72.6% | December 2025 |
| Human baseline | ~72.36% | - |
Since launching the first framework at 20.6% on OSWorld just a year ago, Agent S2 raised the state of the art to 48.8%, and Agent S3 pushed performance to 69.9%, approaching human-level performance.
The technical trick that got Agent S3 over the line matters. Agent S3 introduces Behavior Best-of-N (bBoN), the first wide-scaling framework for computer-use agents through this method. Instead of relying on a single agent run, bBoN selects from multiple rollouts and picks the best outcome. The 72.6% result is not a single lucky run - the best result at 72.6% using GPT-5 and Opus 4.5 surpasses human performance at 72.36%, highlighting that the judge can boost existing methods to reach human-level capability.
The efficiency problem the benchmark score doesn't capture
State-of-the-art systems have focused solely on improving accuracy on leading benchmarks. However, these systems are practically unusable due to extremely high end-to-end latency - tens of minutes for tasks that typically take humans just a few minutes to complete.
A June 2025 paper, OSWorld-Human, measured this directly. Evaluation of real-world computer tasks found that computer-use agents can take tens of minutes to complete a task. For instance, changing the line spacing of two paragraphs in a document to double-spaced takes 12 minutes for a computer-use agent. For a human user with introductory computer experience, this task should take under 30 seconds.
The source of the slowdown is well-understood: large model calls for planning, reflection, and judging account for most of the overall latency, and as an agent uses more steps to complete a task, each successive step can take 3× longer than steps at the beginning.
Even the highest-scoring agents on OSWorld take 1.4-2.7× more steps than necessary.
This is a different failure mode from accuracy. The agent eventually gets it right. It just does it through a long, expensive chain of model calls that would be agonizing to watch in a live workflow - and would cost real money per task at production scale.
Open source vs. closed: where the computer use agent gap actually sits
Agent S3 is the first computer use agent to surpass human performance on the OSWorld benchmark. It runs on macOS, Windows, and Linux, and works with models from OpenAI, Anthropic, and open-weight providers. The framework itself is Apache 2.0 - you can deploy it, modify it, and run it commercially.
But the model underneath it in the best-performing configuration is not open. The 72.6% result required GPT-5 and Claude Opus 4.5 as the backbone and judge, respectively. Run the same framework with a smaller open-weight model and you lose meaningful accuracy. With GPT-5 Mini as the backbone instead of GPT-5, performance drops from 69.9% to 60.2% SR - still a 10-point improvement from Behavior Best-of-N, but a real accuracy gap.
This is a pattern that shows up consistently across agent evals: the orchestration logic is open, the sharp results aren't. It doesn't make Agent S3 less interesting - the framework's progress from 20% to 62% in a single run is genuine - but it means the headline number requires frontier API access that carries per-token inference costs at scale.
A teammate operating in Slack, like Beagle, runs in a tighter constraint: it gets one shot at a response, can't run ten rollouts and pick the winner, and needs sub-second latency to feel like a colleague rather than a batched job. That gap between benchmark conditions and production conditions is not a criticism of the research - it's just the translation cost every team has to price in.
Beagle in action#ops, 2:11pmWhat the benchmark trajectory tells you about deploying a computer-use agent today
In August 2026, Sai - Simular's hosted computer use agent built on the ideas in the Agent S repo - reached a 73% success rate on OSWorld 2.0. The benchmark keeps moving. That's the right signal to follow rather than any single score.
Practically, here's what the numbers imply for teams evaluating these tools:
- Accuracy is solved in lab conditions. 72% on OSWorld with ensemble sampling means the hard research problem of screen comprehension and action chaining is cracked. The remaining 28% is mostly multi-app workflow complexity, not fundamental capability limits.
- Latency is not solved. State-of-the-art agents often require tens of minutes to complete tasks that a human expert can perform in seconds or minutes. This disparity creates a bottleneck for interactive workflows and real-time assistance. If your use case is batch-overnight work (scrape, reformat, export), today's agents are already viable. If it's interactive in a live product, you need to account for that 3× per-step slowdown.
- Cost scales with quality. Getting to 72.6% requires running ten trajectories through two frontier models and having a judge pick the winner. At $0.09 input / $0.40 output per million tokens for open-weight inference on the backbone alone, production costs for a single complex task can reach several dollars. For autonomous background tasks, acceptable. For synchronous workplace actions, still hard to justify.
- The open-source framework matters independently. Even without the best-performing backbone model, Agent S3's architecture contributions - the simplified planning loop, the native coding agent - improve accuracy significantly over Agent S2. The framework is worth tracking regardless of which model you plug in.
Computer use agent benchmark: common questions
What is OSWorld and why does it matter?
OSWorld is a benchmark of 369 real computer tasks executed inside live operating system environments, covering Ubuntu, Windows, and macOS. It measures whether an agent can actually complete tasks using keyboard and mouse control in real applications, making it harder to game than static text benchmarks. It's the current standard for evaluating multimodal computer-use agents.
Has any AI agent beaten human performance on OSWorld?
Yes. Agent S3, released by Simular in December 2025, scored 72.6% on OSWorld using Behavior Best-of-N sampling - marginally above the human baseline of 72.36%. The result requires running the agent ten times and using a second model to judge the best trajectory. A single-run baseline scores 62.6%.
Is Agent S3 open source?
The Agent S3 framework is fully open source under Apache 2.0 and available on GitHub. The best benchmark results, however, require access to frontier closed models (GPT-5 and Claude Opus 4.5) as backbone and judge. Open-weight backbones produce meaningfully lower accuracy on the current benchmark.
Why are computer-use agents still slow even when they score well?
Latency comes from repeated large model calls for planning, reflection, and judging at each step. A June 2025 efficiency study found that each successive step can take 3× longer than the first as context grows, and top agents use 1.4-2.7× more steps than a human would need for the same task. Accuracy benchmarks don't penalize latency at all.
What should teams watch next in computer-use agent development?
The open problem is not accuracy - it's inference efficiency. The next meaningful progress will come from agents that reach human-level task completion in human-comparable time, without ten-run ensembles. Watch for distilled smaller models that close the gap between open-weight and frontier performance, and for frameworks that prune the planning-reflection loop without losing accuracy.