TLDR: MULocBench is a new, comprehensive dataset of 1,100 issues from 46 Python projects designed to improve software issue localization. Unlike previous benchmarks that focus mainly on code and pull requests, MULocBench includes diverse issue types, root causes, and locations across code, configurations, and documentation, and considers resolutions via commits and comments. Evaluations using MULocBench show that current state-of-the-art methods and large language models struggle significantly with realistic, multi-faceted issue resolution, achieving less than 40% accuracy even at the file level, highlighting the need for more advanced techniques.
Software projects today are incredibly intricate, often comprising thousands of files ranging from source code to configurations, tests, and documentation. When an issue arises, whether it’s a runtime error, an unexpected result, or even a request for a new feature, the first critical step to resolving it is accurately identifying its location. This could mean pinpointing a specific file, function, or even a line of code. However, current tools and benchmarks designed to help with this ‘issue localization’ have significant limitations.
Existing benchmarks, such as SWE-Bench and LocBench, primarily concentrate on issues resolved through pull requests and focus almost exclusively on code locations. This narrow scope overlooks a vast array of real-world scenarios where issues might be resolved through simple commits or comments, and where the problem might lie in non-code files like configuration settings, documentation, or even third-party libraries.
Introducing MULocBench: A More Realistic Testbed
To bridge this crucial gap, researchers have introduced MULocBench, a new and comprehensive dataset designed to provide a more realistic evaluation environment for issue localization. MULocBench comprises 1,100 issues meticulously gathered from 46 popular Python projects hosted on GitHub. What sets it apart is its remarkable diversity in issue types, underlying causes, the scope of locations involved, and the types of files affected.
The construction of MULocBench involved a three-stage process: first, selecting the top 50 most-starred Python repositories and sampling 200 issues per project; second, filtering these issues to ensure they were closed, resolved, and had clear location information (whether through pull requests, commits, or explicit comments); and finally, extracting detailed location data including project name, file path, class name, function name, and line numbers.
Understanding the Landscape of Software Issues
An empirical analysis of MULocBench reveals a rich tapestry of software problems:
- Issue Types: Issues are categorized into Execution Failures (39.9%), Unexpected Results (23.7%), Enhancement Requests (25.1%), and Usage Questions (11.3%). This broad coverage, especially the inclusion of Usage Questions, is a significant improvement over prior benchmarks.
- Root Causes: Problems stem from Implementation Bugs (34.5%), Design Deficiencies (36.3%), and User-Induced Problems (29.2%). The recognition of user-induced problems highlights that not all issues originate from project flaws.
- Location Scopes: While most issues involve In-Project Files (94.1%), MULocBench also includes Runtime Files (2.2%), Third-Party Files (2.1%), and User-Authored Files (3.0%), reflecting that resolutions can extend beyond the project’s direct codebase.
- Location Types: Resolutions involve Code (80.8%), Test (23.5%), Configuration (15.2%), Documentation (23.5%), and Asset (4.4%) files. This diversity underscores that issue resolution is not solely a coding task.
Current Methods Fall Short
The researchers used MULocBench to evaluate the performance of state-of-the-art localization methods, including retrieval-based (BM25), procedure-based (Agentless), and agent-based approaches (LocAgent, OpenHands). The results were eye-opening: even the best methods achieved less than 40% accuracy at the file level (Acc@5), and significantly lower at class and function levels. This is a stark contrast to the 60%+ accuracy often observed on simpler benchmarks like SWE-Bench Lite, indicating that current techniques struggle to generalize to the complexity of real-world issue resolution.
Large Language Models: A Promising but Limited Future
Given the advanced reasoning capabilities of Large Language Models (LLMs), the study also explored their effectiveness on the full MULocBench, including non-code files. Five LLM-based prompting strategies were tested, ranging from a ‘Closed-Book’ approach (relying solely on prior knowledge) to ‘Location-Hint Guided’ pipelines (providing project structure and file type hints).
While LLMs showed some promise, particularly with contextual support like project structure and location hints, their performance remained modest. The ‘Location-Hint’ strategy, using Claude 3.5, achieved the strongest results, but still only reached 32.5% Acc@5 at the file level, with performance degrading further for class and function levels. This suggests that while LLMs can benefit from project context and hints, they still face substantial challenges in accurately localizing issues across diverse file types and contexts.
The study also found that LLMs were more effective for code-related files and in-project files but struggled with external dependencies or indirect files. Usage Questions and Implementation Bugs were generally easier to localize than Design Deficiencies or Unexpected Results, likely due to clearer textual cues in their descriptions.
Also Read:
- LLM Agents Tackle Real-World Software Compilation Challenges with BUILD-BENCH
- Understanding Why AI Agent Systems Fail: A Deep Dive into Root Causes
The Path Forward
MULocBench serves as a crucial new tool for the software engineering community, offering a more comprehensive and realistic benchmark for evaluating issue localization techniques. The findings from this research highlight the significant limitations of current state-of-the-art methods and LLMs in handling the multifaceted nature of real-world software issues. This underscores an urgent need for further advancements in developing more robust and generalizable localization approaches. You can find the full research paper here: A Benchmark for Localizing Code and Non-Code Issues in Software Projects.
