What it means
Peng et al. (2023) ran a controlled experiment with 95 developers randomized into GitHub Copilot-assisted and unassisted groups. Participants who used Copilot finished an HTTP server task 55.8% faster — a median of 1.2 hours versus 2.7 hours. The authors flagged the task's narrow, greenfield scope as a ceiling on generalizability. METR's July 2025 RCT tested AI agents on 246 real autonomous software tasks across 69 developers in production-adjacent repositories, averaging roughly 3.5 hours each. The result flipped the sign: AI-assisted developers completed tasks 19% slower than unassisted peers, even though those same participants had predicted a 24% speedup before the trial began. Latent Space's 2024 SWE-bench analysis found the best models solved roughly 50% of real GitHub issues by late 2024, up from under 5% in early 2023. On SWE-bench Verified, frontier models passed only 30–38% of cases. MIT Sloan Management Review found that measured productivity gains from controlled pilots dropped by an average of 35 percentage points when replicated in production environments with existing codebases, attributed to context length limits and legacy code unfamiliarity.
What to do
If this is the problem on your desk, start with our [AI & Software Development](/ai-software-development) practice.
What aI Coding Productivity Studies: Key RCTs and Field Experiments?
Five major AI coding productivity studies span a 75-point range in outcomes — from a 55.8% speedup in Peng et al.'s 2023 controlled experiment to a 19% slowdown in METR's 2025 RCT. The divergence tracks directly to task type: tightly scoped greenfield exercises produce large gains; production-adjacent work with long, ambiguous tasks often does not.
Peng et al. (2023) ran a controlled experiment with 95 developers randomized into GitHub Copilot-assisted and unassisted groups. Participants who used Copilot finished an HTTP server task 55.8% faster — a median of 1.2 hours versus 2.7 hours. The authors flagged the task's narrow, greenfield scope as a ceiling on generalizability.
METR's July 2025 RCT tested AI agents on 246 real autonomous software tasks across 69 developers in production-adjacent repositories, averaging roughly 3.5 hours each. The result flipped the sign: AI-assisted developers completed tasks 19% slower than unassisted peers, even though those same participants had predicted a 24% speedup before the trial began.
Latent Space's 2024 SWE-bench analysis found the best models solved roughly 50% of real GitHub issues by late 2024, up from under 5% in early 2023. On SWE-bench Verified, frontier models passed only 30–38% of cases.
MIT Sloan Management Review found that measured productivity gains from controlled pilots dropped by an average of 35 percentage points when replicated in production environments with existing codebases, attributed to context length limits and legacy code unfamiliarity.
What Should Engineering Leaders Actually Do With This Evidence?
Design your pilot around production conditions, not benchmarks. Track cycle time, defect rate, and PR review time from day one. METR's 2025 RCT found AI-assisted developers were 19% slower on real tasks, so internal data matters more than vendor claims.
Run your pilot on production code, not greenfield exercises. METR's July 2025 RCT found developers completed tasks 19% slower with AI assistance in production-adjacent repositories — the opposite of what participants predicted.
Track cycle time per pull request, defect rate, and PR review time from day one. Roughly 40% of engineers in Pragmatic Engineer's 2024 survey reported AI-generated code added net review burden through new bug categories around edge-case handling.
AI coding tools earn their cost on well-scoped, self-contained tasks — documentation, boilerplate, and test-scaffold generation. Simon Willison's 2024 commentary documents 2–4× gains in that zone, with effectiveness degrading as complexity and codebase entanglement increase.
With 20 years building SaaS products, we at EGV have seen productivity claims dissolve when tools meet legacy code. MIT Sloan Management Review found measured gains dropped by an average of 35 percentage points when pilots moved to production. Your internal data will tell you more than any published study.
| Study / Source | Method | Sample | Task Type | Key Productivity Finding | Generalizability Note |
|---|---|---|---|---|---|
| Peng et al. (arXiv, 2023) | Randomized controlled experiment | 95 software developers | Single, well-specified, greenfield HTTP server task (~1.2 hrs vs. ~2.7 hrs median) | 55.8% faster completion with GitHub Copilot vs. control group | Authors flag single scoped task as a ceiling on generalizability |
| METR (arXiv, July 2025) | Randomized controlled trial (RCT) | 69 developers, 246 tasks | Real autonomous software tasks in production-adjacent repositories (~3.5 hrs avg.) | AI-assisted developers completed tasks 19% slower than unassisted; participants had predicted a 24% speedup | Longer, less-specified tasks than benchmark scenarios |
| HBR Field Study (2023) | Field study | 758 consultants using GPT-4 | Tasks inside vs. outside AI capability frontier | +40% performance on in-frontier tasks; −19 percentage points on out-of-frontier tasks vs. unassisted peers | AI can amplify errors when problem exceeds model's reliable competence |
| MIT Sloan Management Review (2024) | Enterprise deployment analysis | Enterprise AI coding pilots (count not specified) | Controlled pilots replicated in production with existing codebases | Measured gains dropped by an average of 35 percentage points vs. controlled pilots | Gap attributed to context length limits and legacy code unfamiliarity |
| Pragmatic Engineer Survey (2024) | Practitioner survey | 700+ software engineers | Daily/weekly AI coding tool use across varied work | ~70% use AI tools daily or weekly; only 25% report meaningful reduction in code-review time; ~40% report new bug categories from AI-generated code | Self-reported; reflects practitioner experience across production settings |
| Simon Willison (2024) | Longitudinal engineering commentary | — | Well-scoped, self-contained tasks under ~2 hours; documentation, boilerplate, test scaffolds | Personal productivity gains of 2–4× observed on scoped tasks; effectiveness degrades as complexity, entanglement, and ambiguity increase | Single practitioner perspective; not a controlled study |
| Latent Space / SWE-bench Analysis (2024) | Benchmark analysis | — | Real GitHub issues (SWE-bench); SWE-bench Verified subset for frontier models | Best models solved ~50% of SWE-bench issues by late 2024 (up from under 5% in early 2023); SWE-bench Verified pass rates typically 30–38% for frontier models | Benchmark design materially shapes reported capability |
| Gartner Survey (2024) | Engineering-leader survey | 1,400+ engineering leaders | Enterprise AI coding assistant adoption; projected custom software development costs | 75% of enterprise engineers projected to use AI coding assistants by 2028 (vs. fewer than 10% in 2022); projected 30% cost reduction in custom software development over same period | Governance and security review costs not yet factored into most ROI models |