AI Coding Productivity Studies Show a 75-Point Gap in Outcomes

Quick Answer

ai coding productivity studies produce sharply conflicting results depending on experimental design, and understanding those differences is essential before drawing conclusions. Peng et al. (2023) ran a controlled experiment with 95 developers and found that those using GitHub Copilot completed a targeted HTTP server task 55.8% faster than the control group. METR's 2025 randomized controlled trial reached the opposite conclusion: experienced developers working on real open-source repositories were approximately 19% slower when using AI assistance, despite predicting a 24% speedup before the trial. The core explanation is task type. The Peng et al. study used a narrow, isolated, controlled task. METR used real open-source repositories with messy context and multi-file dependencies. A 2023 MIT Sloan Management Review analysis reports that productivity gains tend to be largest for junior and mid-level developers on routine tasks, while senior developers handling complex architectural decisions see smaller or negligible gains. Engineering leaders should match study conditions to their actual work before committing budget to AI coding tools.

Comparison of key ai coding productivity studies referenced in this article. Sources: Peng et al. (arXiv cs.SE, 2023); METR (arXiv cs.AI, 2025); MIT Sloan Management Review (2023); Pragmatic Engineer. Results vary widely by task type, developer experience level, and study design.

Quick Answer

AI coding productivity studies show sharply different results depending on experimental design. Peng et al. (2023) found developers using GitHub Copilot completed a targeted HTTP server task 55.8% faster than a control group in a controlled experiment with 95 developers. METR's 2025 randomized controlled trial found the opposite: AI assistance caused experienced developers to work approximately 19% slower on real open-source repositories, counter to participants' own expectation of a 24% speedup.

The gap comes down to task type. The Peng et al. study used a contained, synthetic task. METR used real-world work on existing codebases. Those are not the same thing, and conflating them is how buyers get burned.

Study-by-Study Comparison: What Each Paper Actually Measured

Two studies anchor the debate. Peng et al. (2023) ran a controlled experiment with 95 developers and found GitHub Copilot users completed a targeted HTTP server task 55.8% faster than the control group. METR's 2025 randomized controlled trial tested experienced developers on real open-source repositories and found AI assistance caused approximately 19% slower work on average — the opposite of participants' own forecast of a 24% speedup.

A 2023 MIT Sloan Management Review analysis adds another dimension: productivity gains appeared largest for junior and mid-level developers on routine tasks, while senior developers handling complex architectural decisions saw smaller or negligible gains from current AI coding tools.

Why the Peng et al. Study Found 55.8% Faster Completion

The design choices matter. Developers worked on a single, isolated targeted HTTP server task in a controlled experiment — conditions that remove most real engineering friction found in production work.

We've spent 20 years building and shipping SaaS products, and isolated tasks represent a small fraction of daily engineering time. The Peng et al. number is real, but it measures a narrow slice of the job.

Why the METR Study Found Developers Working 19% Slower

METR's 2025 randomized controlled trial put experienced AI-using developers on real open-source repositories and measured outcomes under a rigorous controlled design — not isolated, purpose-built tasks.

The result surprised even the participants: AI assistance caused developers to work approximately 19% slower on average, compared to the control condition. Those same developers had predicted a 24% speedup before the trial began.

One design choice set the study apart: tasks involved real open-source repositories with all their existing context and multi-file dependencies, rather than isolated, purpose-built exercises.

The study is a preliminary (arXiv cs.AI, 2025) randomized controlled trial and is hard to dismiss. Readers should note it carries a preliminary designation as of its publication date.

What AI Coding Productivity Studies Reveal About Benchmark Scores

SWE-bench Verified scores look impressive, but context matters. Per analysis by Pragmatic Engineer, top frontier models resolve roughly 40–50% of real GitHub issues when working in isolation. That number drops sharply once tasks require multi-file context, test-writing, and integration with existing codebases — conditions most synthetic benchmarks never include.

The Methodological Variables That Explain the Gap

Four variables explain most of the divergence across studies. First, task type: a timed HTTP server exercise has a clean finish line. Real open-source repositories do not — they carry existing code, multi-file dependencies, and messy context.

Developer experience level is the second variable. A 2023 MIT Sloan Management Review analysis found that gains were largest for junior and mid-level developers on routine tasks, while senior developers working on complex architectural decisions saw smaller or negligible gains.

Measurement horizon is the third. A short controlled task captures speed to first working prototype. A trial on real open-source repositories — as in METR's 2025 study — captures performance under the full complexity of production context.

Fourth, agentic mode versus autocomplete. Top frontier models resolve roughly 40–50% of real GitHub issues in isolation, but performance drops when tasks require multi-file context, test-writing, and codebase integration — conditions absent from most benchmarks.

What Engineering Leaders Should Do With Conflicting Evidence

Do not let a single study set your AI tooling policy. The 55.8% speedup Peng et al. (2023) measured came from a narrow, isolated task; the 19% slowdown METR (2025) measured came from real, open-ended repository work. Both numbers are real. Neither one alone tells you what will happen on your codebase.

Run an internal pilot using representative tasks from your actual backlog. Measure cycle time and defect rate, not just raw completion speed.

Also separate your use cases. Line completion and autonomous agents are different interventions. Benchmark pass rates for top models sit around 40–50% on isolated GitHub issues but drop sharply once multi-file context and integration requirements enter the picture.

We have spent 20 years building and scaling SaaS products with our own team, and every company we have served has been a technology company. For a deeper look at how agentic systems fit into structured delivery, see our work on agent systems and delivery pods.

Key Facts

• Controlled Trial: 55.8% Faster: Peng et al. (2023) ran a controlled experiment with 95 developers and found that those using GitHub Copilot completed a targeted HTTP server task 55.8% faster than the control group (arXiv cs.SE, 2023).
• Controlled Trial: 19% Slower: METR's 2025 randomized controlled trial found that AI assistance caused experienced developers working on real open-source repositories to work approximately 19% slower on average than the control condition — counter to participants' own expectations of a 24% speedup (arXiv cs.AI, 2025).
• Benchmark vs. Real-World Gap: Per Pragmatic Engineer, top frontier models resolve approximately 40–50% of real GitHub issues on SWE-bench Verified in isolation, but performance drops substantially when tasks require multi-file context, test-writing, and integration with existing codebases.
• Who Benefits Most: Per a 2023 MIT Sloan Management Review analysis, productivity gains were largest for junior and mid-level developers on routine tasks, while senior developers handling complex architectural decisions saw smaller or negligible gains from current AI coding tools.
• The Outcome Gap: The two anchor studies show results ranging from 55.8% faster (Peng et al., 2023, controlled experiment with 95 developers on a targeted HTTP server task) to approximately 19% slower (METR, 2025, randomized controlled trial on real open-source repositories) — illustrating how widely results vary across task type, developer experience, and study design.

If this is the problem on your desk, talk to us.

Research and drafting assisted by Educated Guess Ventures' Corner Post engine; reviewed and edited by the author.

For a deeper treatment, see our analysis of onshore vs offshore software development total cost of ownership.