The Role of AI in Software Development: Surprising Findings

In a recent exploration, software developers entered a controlled environment expecting artificial intelligence to significantly reduce their project completion times. Contrary to their expectations, the use of AI tools resulted in a 19% increase in completion time, despite participants reporting a subjective sense of increased efficiency. This paradox emerging from a 2025 study by the nonprofit Model Evaluation and Threat Research (METR) has sparked lively discussions regarding the real impact of AI on professional coding practices.

The METR Study Overview

The METR study, detailed in a paper published on arXiv and summarized on their official blog, involved 16 developers with an average of five years’ experience working on substantial open-source repositories—each boasting over 22,000 stars and a million lines of code. The participants were tasked with 246 authentic assignments ranging from bug fixes to feature enhancements. Half of the developers utilized AI tools like Cursor Pro, paired with either Claude 3.5 or 3.7 Sonnet, while the other half worked without assistance. Initially, developers predicted they would achieve a 24% increase in speed due to AI, and afterward, they self-reported gains of approximately 20%. However, the results revealed a stark divergence from these expectations.

Surprising Insights from Participants

Researchers Joel Becker and Nate Rush, who led the METR effort, were taken aback by the findings. “Many developers noted that even when the AI outputs proved somewhat useful, they still had to invest considerable time into refining the resulting code to make it fit for their projects,” Rush explained to Fortune. Time logs and screen recordings highlighted the inefficiencies: the processes of prompting, waiting, addressing inaccuracies, and integrating code into existing systems consumed more time than the AI had purportedly saved.

Perception vs. Reality

This discrepancy between perceived and actual productivity mirrors a broader trend. Despite explicit instructions to deploy AI only when beneficial, many participants overestimated its advantages. Developer Philipp Burckhardt reflected in a blog post, “While I like to believe that using AI for my tasks didn’t hurt my productivity, it’s plausible that it didn’t assist me as much as I had hoped or perhaps even hindered my efforts.” The METR authors observed that developers engaged in varying modes of AI usage—normal operations, experimentation, and overreliance—each of which contributed to delays, especially in complex, context-laden tasks.

Significantly, the study noted that tasks where developers possessed extensive prior knowledge experienced the most considerable slowdowns. AI often struggled with project-specific nuances that seasoned developers navigated effortlessly. “Developers have goals that extend beyond mere task completion speed,” Becker articulated to Reuters, emphasizing that many users, including those involved in the study, continued to use AI tools for a more seamless workflow, akin to editing an essay.

Contrasting Perspectives from Other Studies

The METR findings stand in stark contrast to more optimistic reports. A GitHub-Microsoft study indicated that developers completed a JavaScript HTTP server 55.8% faster using Copilot, as stated in a 2023 arXiv paper. Larger field studies conducted at Microsoft, Accenture, and among Fortune 100 companies reported a 26% increase in task completion enabled by Copilot, with junior developers seeing gains of 27-39% compared to 8-13% for senior developers.

Understanding the Differences

What accounts for these differing outcomes? METR emphasized the importance of mature projects compared to simpler benchmarks or greenfield tasks. “At first glance, METR’s results seem to contradict other benchmarks… which often measure productivity in terms of total lines of code or number of discrete tasks,” noted Ars Technica. A study by Qodo echoed that verification overhead could diminish productivity gains, while workforce data from Denmark showed a mere 3% improvement, as reported by The Register.

The cognitive burden is substantial: Developers reportedly spent 34.3% of their sessions verifying Copilot’s suggestions alone, as highlighted by Google DeepMind’s Paige Bailey in internal visuals posted on X. Issues related to review fatigue and context-switching have also been identified, aligning with American Psychological Association findings on the costs of task-switching.

Economic Expectations vs. Real-World Applications

High expectations for AI’s impact falter under scrutiny. PwC anticipated a 15% increase in U.S. GDP by 2035, while Goldman Sachs forecasted a 25% rise in productivity. However, MIT found that only 5% of 300 AI deployments yielded rapid revenue acceleration, according to a report by Harvard Business Review. Anders Humlum from the University of Chicago remarked, “In reality, many tasks are not as straightforward as merely typing into ChatGPT. Experts often possess extensive experience that is highly beneficial.”

Examining Constraints and Limits

The METR team cautioned about the study’s limitations—small sample size, novel tools for many participants (56% of whom were new to Cursor), and the absence of junior developers or unfamiliar codebases. One participant with over 50 hours of experience using Cursor reported improved speed, suggesting that while initial use may slow down performance, proficiency may mitigate these effects. Daron Acemoglu at MIT estimated that AI optimally aids only about 4.6% of tasks in the U.S. Discussions on X, including posts like Vladimir’s, contrasting Copilot’s 55% lab successes with METR’s findings, raise questions about the pressures facing teams amidst enormous hype.

More recent data further tempers expectations. A study published in Science indicated that AI coding functions will account for 29% of U.S. tasks by early 2025, leading to a 3.6% boost in productivity, primarily for experts willing to engage actively, according to TechXplore. Trials by Anthropic confirmed that while Claude enhances task performance, it may also diminish skills and collaboration among developers.

Adapting to New Realities

As the industry evolves, practitioners are re-evaluating their approaches. “AI coding tools introduce added cognitive load and context-switching that disrupt developer productivity,” Augment Code reported based on METR findings. Cerbos recommended focusing on objective metrics rather than subjective perceptions, citing a stark 43-point gap between expected and actual outcomes. Threads on Reddit’s r/programming reveal frustrations regarding AI errors that prolong fixing time compared to manual coding efforts.

Importance of Training and Strategy

Effective training can make a difference: While live sessions with Cursor did not eliminate slowdowns, consistent practice might yield improvements. The capabilities of AI tools are evolving, with METR noting that AI can double the efficiency of long-term tasks every seven months. User @tekbog on X cautioned against the risks of AI leading to suboptimal practices, while Martin Fowler and Birgitta Böckeler expressed concerns about increased fatigue and the amplification of bad habits.

Conclusions and Recommendations

The insights gleaned from the METR study offer valuable lessons for leaders in the field: it is advisable to pair AI selectively with junior developers or on unfamiliar code; rigorous reviews should be enforced; and objective performance metrics should be tracked. As noted by Reuters, productivity gains do not apply universally, particularly for experienced developers within their domains. Ultimately, many continue using tools like Cursor for the enjoyment they derive from coding, shifting their focus beyond mere speed and output.