TLDR: A new research paper introduces “AutoEmpirical,” an LLM-based system designed to automate the labor-intensive process of analyzing software faults. By breaking down fault analysis into three stages—research definition, data preparation, and fault analysis—the system significantly speeds up the process, reducing weeks of manual work to just two hours. While LLMs show strong performance in identifying fault-related issues and bug symptoms, they currently struggle with accurately classifying the root causes of bugs, highlighting areas for future development in context-aware reasoning and tool integration.
Understanding and analyzing software faults is a critical but often arduous task in software development and maintenance. Traditionally, this process involves extensive manual effort from experts, including collecting, filtering, and investigating potential faults. This labor-intensive and time-consuming approach creates significant bottlenecks, hindering large-scale studies of complex software systems and slowing down iterative empirical research.
A new research paper, titled “AutoEmpirical: LLM-Based Automated Research for Empirical Software Fault Analysis,” explores a promising solution: leveraging Large Language Models (LLMs) to automate and accelerate these empirical studies. Authored by Jiongchi Yu, Weipeng Jiang, Xiaoyu Zhang, Qiang Hu, Xiaofei Xie, and Chao Shen, the paper introduces a novel pipeline designed to streamline the entire fault analysis workflow.
The AutoEmpirical Pipeline: A Three-Stage Approach
The researchers decompose the empirical software fault study into three key phases, each assigned to collaborating LLM agents:
1. Research Objective Definition: In this initial stage, LLMs assist in defining research objectives, selecting appropriate software repositories, and determining the scope of analysis. While LLMs successfully identified key projects, the study noted a tendency for them to overlook a broader range of relevant third-party libraries, suggesting a need for more comprehensive project selection capabilities.
2. Data Preparation (Fault-Related Issue Selection): This phase focuses on identifying and selecting fault-related issues or commits from repositories for in-depth analysis. The LLM agent makes a binary decision—fault-related or non-fault—based on the textual and metadata content of the issue. The evaluation showed promising performance, with LLMs achieving high accuracy in distinguishing bug-related issues, though they sometimes included non-security-related issues like testing issues.
3. Fault Analysis (Taxonomy-Based Classification): The final stage involves classifying each selected issue’s symptom and root cause using a predefined taxonomy. LLMs are provided with complete hierarchical structures and detailed definitions for both symptom and root cause categories. This fine-grained evaluation tested the models’ ability to map textual inputs into a fixed taxonomy.
Key Findings and Performance
The study evaluated five state-of-the-art LLMs: GPT-4o, GPT-o3, Claude 3.7, Gemini-2.5, and DeepSeek-V3. The results demonstrated a substantial improvement in efficiency, with the LLM-driven pipeline completing the analysis in approximately two hours, a remarkable 20x speed-up compared to the weeks typically required for manual effort.
In terms of accuracy, LLMs showed strong capabilities in identifying bug-related issues, with Gemini-2.5-Flash demonstrating the best performance in this stage. For classifying bug symptoms, OpenAI GPT-4o exhibited outstanding performance, significantly outperforming other models. However, a notable limitation emerged in the classification of bug root causes. Even with a predefined taxonomy, LLMs generally struggled, with Gemini achieving the best performance at only 49% accuracy. This limitation is attributed to the lack of detailed code and contextual information within the issue reports, which are crucial for accurate root cause analysis.
Also Read:
- Assessing Code Quality: A Deep Dive into LLM-Generated Code Smells
- Enhancing Automated Program Repair with Intelligent Filtering Policies
Future Directions for Automated Research
The preliminary investigation highlights both the immense potential and current limitations of LLM-driven automation for empirical fault analysis. The researchers outline a detailed agenda for future work, including:
- Collecting comprehensive, diverse, and high-quality datasets from top-tier software engineering venues to create robust benchmarks.
- Evaluating LLMs’ capability to autonomously construct, apply, and refine fault taxonomies, moving beyond reliance on predefined frameworks.
- Developing a fully automated, end-to-end agent system that incorporates context-aware reasoning (e.g., dynamically retrieving and analyzing source code), integrates with software engineering tools (like debuggers and version control systems), and exhibits evolving autonomy to recognize limitations and seek clarification.
In conclusion, this pioneering work establishes a foundation for scalable automated studies in software fault analysis. While current LLMs offer significant efficiency gains in identifying fault data, improvements in fine-grained taxonomy classification and contextual understanding are essential for achieving fully automated, end-to-end software fault analysis. You can read the full research paper here.
