TLDR: RepoDebug is a novel, multi-task, and multi-language dataset designed to evaluate Large Language Models (LLMs) on complex, repository-level code debugging. It features 22 error subtypes across 8 programming languages and assesses bug identification, localization, and automatic program repair. Evaluations on 10 LLMs, including Claude 3.5 Sonnect, show that current models still face significant challenges in these realistic scenarios, particularly with multiple errors, lower-level languages, and longer codebases, highlighting the need for further research.
Large Language Models (LLMs) have shown impressive capabilities in various coding tasks, including automatically repairing program errors. This ability is crucial for developers, as it can significantly reduce the time and effort spent on debugging and improve overall efficiency. While many datasets have been developed to evaluate LLMs’ debugging skills, most have focused on fixing errors within single functions. This approach, however, doesn’t fully capture the complexities of real-world software development, where errors often span across multiple files and involve intricate project-level interactions.
Recognizing this gap, a new research paper introduces RepoDebug, a groundbreaking dataset designed to evaluate LLMs on more realistic, repository-level code debugging scenarios. This dataset is unique because it incorporates multiple tasks, supports various programming languages, and features a diverse range of error types.
What is RepoDebug?
RepoDebug is a comprehensive benchmark that addresses the limitations of previous debugging datasets. It includes 22 distinct subtypes of errors, systematically categorized into four main types: syntax errors, reference errors, logic errors, and multiple errors. The dataset supports 8 commonly used programming languages: C, C#, Go, Java, JavaScript, Python, Ruby, and Rust. Furthermore, it evaluates LLMs across three critical debugging tasks: Bug Identification (determining the type of error), Bug Localization (pinpointing the exact location of the error), and Automatic Program Repair (fixing the error).
The creation of RepoDebug involved collecting data from 63 GitHub repositories, all created after 2022 to minimize data leakage. Bugs were meticulously introduced into code files using an Abstract Syntax Tree (AST) parser, allowing for precise control over error placement and ensuring that the errors genuinely affect the project’s functionality. This AST-based approach ensures that the injected bugs are diverse, realistic, controllable, observable, and representative of actual developer mistakes.
How LLMs are Evaluated
The researchers evaluated 10 different LLMs, including popular closed-source models like GPT-4o, GPT-4o-mini, and Claude 3.5 Sonnect, as well as several open-source models such as Qwen2.5 Coder, StarCoder2, Deepseek-Coder-V2, Code Llama, and Deepseek-R1. The evaluation used four key metrics: Bug Identification (BI) accuracy, One Bug’s Location (OBL) accuracy, All Bugs’ Location (ABL) accuracy, and Automatic Program Repair (APR) measured by Edit Similarity (ES), Exact Match (EM), and Pass@1.
Also Read:
- ReCode: Enhancing AI’s Code Repair Capabilities with Smart Retrieval
- The Loong Project: Advancing AI Reasoning with Scalable Synthetic Data and Verification
Key Findings and Challenges
The experimental results from RepoDebug highlight significant limitations in the current generation of LLMs when it comes to repository-level code debugging. Even the best-performing model, Claude 3.5 Sonnect, still struggles to perform well in these complex scenarios. Here are some of the key observations:
- LLMs exhibit substantial performance differences across programming languages. They generally perform better on high-level languages like Java and JavaScript but struggle with lower-level or statically typed languages such as C and Rust. This could be due to the inherent complexity of low-level languages and a bias in training data, which often contains more high-level language code.
- The type of error significantly influences model performance. Syntax errors were found to be the easiest to detect and fix, while multiple errors (instances with 2 to 4 bugs) proved to be the most challenging. This suggests that LLMs have limited ability to understand code semantics and handle multiple error types simultaneously.
- An increase in code length negatively impacts model performance. Models performed significantly worse on longer code segments (over 500 tokens) compared to shorter ones, indicating that processing long-range contextual information remains a major challenge for LLMs.
- Qualitative analysis revealed that models are often better at generating plausible fixes than accurately localizing the underlying errors. They also tend to over-identify errors, sometimes introducing spurious corrections unrelated to the actual bugs.
This research underscores that while LLMs show promise in code debugging, there’s still a long way to go before they can reliably handle the intricacies of real-world, repository-level projects. The RepoDebug dataset is a valuable contribution to the field, providing a robust benchmark to drive future research and development in this critical area of AI in software engineering. For more details, you can refer to the original research paper.
