TLDR: A new dataset, AIDev, reveals how autonomous AI coding agents like OpenAI Codex and GitHub Copilot are actively contributing to open-source projects. While these agents significantly accelerate code submission and review, their pull requests have lower acceptance rates than human-authored ones, especially for complex tasks. Documentation is an AI strength, but authorship attribution and the long-term quality of AI-generated code remain key challenges for the evolving “SE 3.0” era.
Software engineering is undergoing a significant transformation, moving into an era dubbed “SE 3.0,” where artificial intelligence (AI) is no longer just an assistant but a full-fledged teammate. This new paradigm sees autonomous, goal-driven AI systems collaborating directly with human developers in real-world software development workflows.
At the forefront of this shift are autonomous coding agents. These sophisticated AI systems are now actively involved in initiating, reviewing, and evolving code on a massive scale within the open-source ecosystem. Tools like OpenAI Codex, Devin, GitHub Copilot, Cursor, and Claude Code are becoming routine contributors to software projects.
To better understand how these AI teammates operate in practice, researchers have introduced AIDev, the first large-scale dataset specifically designed to capture their real-world activities. This extensive dataset spans over 456,000 pull requests authored by these five leading autonomous agents across 61,000 repositories and involving 47,000 developers. AIDev provides an unprecedented foundation for studying the integration of AI teammates into modern software development, offering concrete, structured, and open data for future research.
One of the key findings from the AIDev dataset is that while autonomous coding agents can dramatically accelerate code submission, their pull requests (PRs) are accepted less frequently than those submitted by human developers. This reveals a notable gap between how these agents perform in controlled benchmarks and their actual utility and trustworthiness in real-world scenarios. For instance, agents often complete tasks much faster than humans; GitHub Copilot, for example, finishes 75% of its PRs in under 18.5 minutes, a speed far exceeding typical human issue resolution times. Similarly, PRs from OpenAI Codex are reviewed and merged significantly quicker, sometimes in just 18 minutes, compared to several hours for human-authored PRs. However, this speed doesn’t always translate to higher acceptance rates, especially for complex tasks like feature development and bug fixing.
Interestingly, documentation tasks emerge as a clear strength for autonomous coding agents. Both OpenAI Codex and Claude Code show high acceptance rates for documentation-related PRs, often outperforming human baselines. This is likely due to these tasks aligning well with the natural language generation capabilities of large language models that power these agents.
The study also highlights evolving code review dynamics. While human reviewers still play a dominant role, there’s a noticeable shift towards hybrid human-bot collaboration, particularly with GitHub Copilot. It was observed that review bots often originate from the same provider as the coding agent, creating streamlined but potentially biased review loops. Furthermore, the research points out a current limitation: AI-generated code tends to prioritize sheer output volume over structural complexity, meaning it often introduces simpler changes compared to human contributions.
A crucial aspect for accountability and transparency is authorship attribution. The study found that while some agents like Devin, GitHub Copilot, and Cursor explicitly indicate their authorship, others like OpenAI Codex and Claude Code often attribute contributions to human developers or provide no attribution at all. This lack of clear ownership can complicate debugging and responsibility assignment in the long run.
Also Read:
- GenAI’s Role in Automotive Software: From Concept to Code
- AI for Debugging: A Reality Check on Verified Bug Fixes
Looking ahead, the research paper envisions “SE 3.0” as a future where software repositories become interactive training environments for AI agents, allowing them to continuously learn and improve through real-world feedback. It advocates for dynamic benchmarking and “living leaderboards” that assess agents based on their effective integration into live projects, rather than just static test cases. The authors also suggest a rethinking of traditional software engineering methodologies like Agile and DevOps to accommodate the unique capabilities and challenges introduced by always-active, autonomous AI teammates. The AIDev dataset, publicly available at https://github.com/SAILResearch/AI_Teammates_in_SE3, is intended to be a living resource to drive this new generation of research into AI-native software engineering workflows.
