Experienced developers working on real open-source repositories took 19% longer to complete tasks when they were allowed to use AI tools. Then they estimated, on average, that the tools had made them 20% faster. That gap - 39 percentage points between perceived and measured performance - is the most important data point in the agentic coding debate right now, and most engineering leaders are not acting on it.
A randomized controlled trial posted in July 2025 by METR found that experienced open-source developers took 19% longer to complete real coding tasks when AI tools were allowed. After the study, those same developers estimated, on average, that AI had made them 20% faster. This was not a benchmark. The study design was unusually close to real work - developers were not asked to solve toy problems. They worked on issues from open-source repositories they already knew well, with an average of about five years of prior experience with the projects involved.
The industry's response has largely been to explain the finding away. The models have improved since early 2025. The tasks were unusually hard. Sixteen developers is a small sample. All true. But the finding rhymes with a pattern showing up across multiple datasets, and the explanation teams reach for - "our engineers feel faster, so they must be" - is precisely the trap the METR data exposes.
Why agentic coding tools widen the seniority gap
Senior engineers extract disproportionate value from agentic tools. They have the codebase context, the architectural judgment, and the domain knowledge to steer AI output, catch its failures, and integrate its work into systems that will hold up under production load. Senior engineers get a compounding advantage: they have the codebase context, the architectural judgment, and the domain knowledge to steer AI output, catch its failures, and integrate its work. AI amplifies what they already know.
Junior engineers get something different. They generate more code - 92% of US developers now use AI coding tools daily, and among Y Combinator's Winter 2025 cohort, 21% of companies report codebases that are 91% AI-generated. But generation volume is not the same as judgment. A junior engineer uses the same tools, generates the same volume of code, and passes the same CI checks. But when the AI produces an approach she doesn't recognize, she searches Stack Overflow to confirm it looks reasonable, then merges it if nothing obviously contradicts it. Both engineers show green DORA metrics. Both have strong sprint velocity. The gap between them isn't in the numbers.
That invisible gap becomes visible in code quality. Security vulnerabilities were 2.74 times more common in AI-co-authored pull requests, according to a December 2025 analysis of 470 PRs. And in churn: a developer using Copilot or Cursor can generate 3-5x more lines per session, but raw volume says nothing about whether that code survives its first month in production. GitClear's 2024 data showed code churn rising from a 3.3% baseline in 2021 to 5.7-7.1% by 2024-25. More code, faster, is not the same as more value, faster.
The metric everyone is using is now actively misleading
PR count, commit frequency, and lines of code were already imperfect proxies. With AI coding assistants, they are actively misleading. The problem is that agentic tools make these numbers go up regardless of whether quality improves. A developer who generates three times as many lines in a session looks more productive by every traditional metric, even if her senior teammate now spends an extra hour each day reviewing and correcting the output.
One paper concluded that apparent productivity gains often shifted work from junior contributors to senior maintainers responsible for production quality control. This is the second-order effect that most engineering velocity dashboards cannot see. METR's trial found that experienced developers spent more time debugging AI-generated code than writing it themselves, because they had to verify complex logic and architectural decisions that AI could not fully understand.
Daily AI users produce around 4x the raw output of non-users, but measured against their own output a year earlier, the real productivity gain is only about 12%. That 4x-to-12% compression is the core of the measurement problem: teams see the raw volume signal and stop there. The 12% figure requires tracking the same developers over time against real outcomes - not against other developers using the same tools simultaneously.
The bottleneck has moved to review, and most teams haven't noticed
Code review used to work because of a happy accident of relative speed: a senior engineer could read code faster than a junior could write it, so review kept pace without anyone designing it to, and the team absorbed how the system fit together as a side effect of reading each other's diffs. A lot of that was not deliberate.
Agentic tools have broken that ratio. A junior engineer using Claude Code or Cursor can now produce code faster than a senior can meaningfully review it. The review queue lengthens. Seniors, under pressure, approve more than they inspect. And because the code looks syntactically clean - AI does not write sloppy-looking code, it writes plausible-looking code - the failures are architectural or security-level, the kind that surface in production six months later.
The people designing systems and making big architectural decisions are using AI least, while juniors cranking out code use it most. The type of work AI is doing is more on the periphery - small feature PRs, minor fixes - than at the core architectural level. This limits the impact on major outcomes.
The steelman for the optimistic view is real: some teams have found the right workflow. Spotify's Co-CEO Gustav Soderstrom stated during the Q4 2025 earnings call that senior engineers have shifted entirely from writing syntax to generating and supervising AI-produced code since December 2025. This transition relies on Anthropic's Claude Code and an internal system named Honk - a background coding agent that integrates with Spotify's Fleet Management platform to execute source-to-source transformations across repositories via MCP. The workflow allows developers to trigger bug fixes or feature additions through Slack, receive compiled app versions for review, and merge changes to production without manual coding. That is a genuine organizational redesign around AI - not a Cursor subscription bolted onto an unchanged review process.
Beagle in action#eng-team, 10:22amWhat to measure instead
A better measurement stack for agentic coding tools:
Code churn rate - the percentage of code rewritten within 14-30 days. A healthy AI-assisted team keeps this below 1.3× the pre-AI baseline. AI-assisted code inherently has higher turnover, but a well-managed team keeps the ratio below 1.3×. If your 30-day turnover exceeds 18%, audit your AI-generated PRs for review thoroughness.
Review turnaround per PR - if this is rising while PR volume rises, your senior engineers are becoming a bottleneck. The throughput gain is happening in generation; the constraint has shifted to review.
Security finding rate in AI-co-authored PRs - track separately from human-authored. The 2.74× finding from the December 2025 analysis suggests this deserves its own dashboard column.
Per-engineer total AI tool cost - teams using agentic tools spend $200-$2,000+ per engineer per month in token costs on top of seat licenses. Claude Code can cost $200-$2,000+ per engineer per month in token spend - not the $30-60 seat license that older estimates assumed. That range means the ROI calculation needs to be done at the team level, not just assumed from aggregate productivity surveys.
None of this argues against agentic coding tools. The 12% real productivity gain is real. The 3.6 hours per week saved is real. The Spotify workflow is a plausible direction for teams large enough to build internal tooling around it. The argument is narrower: the metrics most teams use to evaluate these tools are measuring the wrong thing, and the gap between what they show and what is actually happening is large enough - 39 percentage points in the METR data - to matter for headcount decisions, tool budgets, and where teams invest in review infrastructure.
Agentic coding tools productivity: common questions
Do agentic coding tools actually improve developer productivity?
It depends on experience level and task type. Follow-up studies after the initial METR result showed improvement, with repeat participants achieving an 18% speedup as they learned how to work more effectively with AI tools. Gains are strongest on isolated, narrowly scoped tasks and for senior engineers who can validate AI output. Mature codebases with complex dependencies see smaller or negative gains initially.
Why did METR find that AI tools slowed developers down?
With AI tools, developers spent less time actively coding and searching for information, but more time on verification and correction. They felt more productive because they were writing less code manually. But the total time from task start to completion actually increased. The overhead of prompting, reviewing, and correcting AI output outweighed the time saved on generation for experienced engineers working in complex, familiar codebases.
What is a healthy code churn rate for teams using AI coding tools?
Code churn - the share of code rewritten within two to four weeks - is a leading indicator of AI-assisted code quality. The pre-AI baseline was 3.3% (GitClear, 2021). By 2024-25, this had risen to 5.7-7.1% across the industry. Keeping churn below 1.3× your pre-AI baseline is a reasonable target; beyond that, review thoroughness is likely the root cause.
Are senior or junior engineers getting more value from AI coding tools?
Senior engineers get compounding returns; junior engineers get volume without the judgment to govern it. A July 2025 survey of 791 professional developers by Fastly found that senior engineers with ten or more years of experience are nearly 2.5 times more likely than junior engineers (zero to two years) to have more than half their shipped code AI-assisted. The seniors are both generating and validating more. The gap between them is growing, not shrinking.
How much do agentic coding tools actually cost per engineer?
Healthy ROI on AI coding tools runs 2.5-3.5× on average, with top-quartile organizations reaching 4-6×. Agentic tools like Claude Code can cost $200-$2,000+ per engineer per month in token spend, making total AI tool cost $200-$600/month per engineer on average
- a number that makes the ROI case less obvious than vendor-sponsored surveys suggest when you model it at team scale.