TLDR: A recent study by AI research nonprofit METR found that, contrary to expectations, using AI coding assistants like Cursor increased task completion time by 19% for experienced software developers working on familiar open-source projects. Developers often spent more time correcting AI suggestions than gaining efficiency.
A groundbreaking study conducted by the AI research nonprofit METR has unveiled a surprising finding: artificial intelligence tools, often touted as productivity boosters, are actually slowing down experienced software developers. The study, carried out earlier this year, revealed that seasoned developers working on familiar open-source codebases experienced a 19% increase in task completion time when utilizing AI coding assistants, such as Cursor.
This outcome stands in stark contrast to the developers’ initial expectations. Prior to the study, participants anticipated a significant acceleration, estimating that AI would decrease their task completion time by an average of 24%. Even after completing their assignments with AI assistance, many developers remained convinced they had reduced their work time by approximately 20%. However, METR’s rigorous analysis demonstrated the opposite effect.
The lead authors of the study, Joel Becker and Nate Rush, expressed their astonishment at the results. Rush, in particular, had initially predicted a ‘2x speed up,’ a common sentiment reflecting the widespread belief in AI’s capacity to supercharge human productivity. This belief has driven substantial investments in AI-powered development tools across the tech industry.
The primary reason identified for this slowdown was the time developers had to dedicate to reviewing and correcting the suggestions generated by the AI models. While AI can quickly produce code, ensuring its accuracy, efficiency, and compatibility within complex, established codebases requires significant human oversight from experienced professionals.
This research challenges the prevailing narrative that AI universally enhances developer output, especially for those intimately familiar with large, intricate projects. The study suggests that while AI might offer benefits in other scenarios or for less experienced developers, its application in familiar, nuanced environments can introduce unforeseen overhead.
Despite the measured slowdown, the study noted that many participants, including the authors themselves, continue to use AI tools. They cited factors such as the ease of use and a reduction in the perceived ‘chore-like’ aspects of development as reasons for their continued adoption. This indicates that while raw productivity might be impacted in specific contexts, other qualitative benefits of AI tools might still hold value for developers.
Also Read:
- Experimental Studies Confirm Generative AI’s Significant Impact on Workforce Productivity
- Bill Gates Predicts AI Will Transform Workforce, But Coding to Remain Exclusively Human for Centuries
The findings arrive amidst broader discussions about AI’s impact on the workforce, including concerns about job displacement. Experts like Dario Amodei, CEO of Anthropic, have suggested that AI could replace entry-level coding positions within the next one to five years, a sentiment echoed by recent layoffs in major tech companies like Microsoft, Google, and IBM, which have cited a shift towards AI as a contributing factor.
