TLDR: A study on experienced open-source developers found that using early-2025 AI tools like Cursor Pro and Claude 3.5/3.7 Sonnet unexpectedly increased task completion time by 19%, contradicting both developer and expert predictions of significant speedup. Factors contributing to this slowdown include developer over-optimism, high familiarity with complex codebases, and the AI’s low reliability and lack of implicit repository context. The findings highlight a disconnect between perceived and actual AI impact in real-world, complex software development settings.
In the rapidly evolving landscape of artificial intelligence, the promise of AI tools to revolutionize various industries, especially software development, has been a dominant narrative. Many anticipate significant boosts in productivity, faster task completion, and an overall more efficient workflow. However, a recent randomized controlled trial (RCT) conducted by Model Evaluation & Threat Research (METR) presents a surprising counter-narrative, suggesting that early-2025 AI tools might actually slow down experienced open-source developers.
The study, titled “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity,” delved into the real-world impact of AI by observing 16 experienced open-source developers. These developers, with an average of five years of prior experience on their respective mature projects, were assigned 246 tasks. Crucially, each task was randomly designated to either allow or disallow the use of AI tools. When AI was permitted, developers primarily utilized Cursor Pro, a popular code editor, and advanced language models like Claude 3.5 and 3.7 Sonnet.
The Unexpected Outcome
Before the study began, developers were optimistic, forecasting that AI would reduce their task completion time by an average of 24%. After completing the study, their post-hoc estimates remained similarly positive, predicting a 20% reduction in time. Even experts in economics and machine learning shared this optimism, forecasting a substantial 38-39% decrease in completion time. Yet, the actual findings painted a starkly different picture: allowing AI tools increased task completion time by 19%. In essence, AI tooling unexpectedly slowed the developers down.
Understanding the Slowdown
To unravel this surprising result, the researchers meticulously analyzed screen recordings and collected extensive data. They identified several key factors likely contributing to this slowdown:
-
Over-optimism about AI usefulness: Developers’ initial belief in AI’s benefits led them to potentially overuse the tools, even when they weren’t proving efficient.
-
High developer familiarity with repositories: The developers were highly skilled and deeply familiar with their codebases, often having years of experience and thousands of commits. In such scenarios, their existing expertise left less room for AI to provide significant, novel assistance.
-
Large and complex repositories: The study involved large, mature open-source projects (averaging 1.1 million lines of code). AI tools appeared to struggle more in these complex environments, sometimes introducing errors or making irrelevant changes that required additional time to fix.
-
Low AI reliability: Developers accepted less than 44% of AI-generated code suggestions. A significant portion of their time (around 9%) was spent reviewing and cleaning up AI outputs, even when accepted, indicating that the AI often produced code that didn’t meet the high quality standards of these projects.
-
Implicit repository context: Experienced developers rely on a wealth of undocumented, tacit knowledge about their codebase. AI systems, lacking this deep contextual understanding, often generated less useful or even counterproductive suggestions.
The study also explored other potential factors, such as unfamiliar development environments or learning curves with AI tools, but found them unlikely to be primary contributors to the observed slowdown. The robustness of the slowdown effect across various analyses suggests it’s not merely an experimental artifact.
Also Read:
- New Research Uncovers Security Concerns in AI Coding Agents
- Streamlining Proto-Persona Creation with Generative AI: A Case Study on Efficiency and User Acceptance
Implications and Future Outlook
The findings of this research, available at arXiv.org, underscore a significant disconnect between the perceived and actual impact of AI on experienced developer productivity in specific, real-world settings. While the results do not imply that current AI tools are universally unhelpful—they may still offer substantial benefits in different contexts, such as greenfield projects or for less experienced developers—they highlight the importance of rigorous field experiments over relying solely on forecasts or surveys.
The authors acknowledge that future AI models with higher reliability, lower latency, or improved elicitation strategies (e.g., better prompting or domain-specific fine-tuning) could potentially yield positive speedup in this setting. Indeed, preliminary evidence suggests that autonomous AI agents can make meaningful progress on some tasks, even if they don’t fully meet all requirements. This study serves as a crucial data point in understanding the nuanced and complex relationship between advanced AI and human productivity in professional software development.
