TLDR: TriagerX is a new bug triaging framework that uses a dual-transformer architecture to recommend developers and components. It combines a Content-based Ranker (CBR) that analyzes bug report text with an Interaction-based Ranker (IBR) that considers developers’ historical contributions and their recency. TriagerX significantly outperforms existing methods, improving accuracy by over 10% for developer recommendations and up to 54% in some cases, and has been successfully deployed in an industrial setting.
In the fast-paced world of software development, identifying and assigning reported bugs to the right developer or team is a critical, yet often challenging, task. This process, known as bug triaging, can significantly impact a project’s efficiency and the speed at which issues are resolved. Traditional methods and even modern machine learning models have faced limitations, struggling with the nuanced language of bug reports and often overlooking the valuable history of developer interactions.
A new research paper introduces TriagerX: Dual Transformers for Bug Triaging Tasks with Content and Interaction Based Rankings, a novel framework designed to overcome these challenges. Developed by Md Afif Al Mamun, Gias Uddin, Lan Xia, and Longyu Zhang from the University of Calgary, York University, and IBM Canada, TriagerX offers a sophisticated approach to automate bug assignments more accurately and reliably.
Addressing the Limitations of Existing Systems
Previous bug triaging systems, including those based on advanced Pretrained Language Models (PLMs), often fall short in two key areas. First, PLMs might not always focus on the most relevant parts of a bug report, leading to less effective recommendations. Second, they frequently ignore the crucial historical interactions developers have had with similar bugs, which human triagers often consider. TriagerX was specifically engineered to tackle these issues head-on.
The Dual-Transformer Advantage
At its core, TriagerX employs a unique dual-transformer architecture. Unlike other state-of-the-art systems that use a single transformer, TriagerX gathers recommendations from two separate transformers. Each transformer contributes insights from its final three layers, creating a robust “content-based ranking” of potential developers. This dual approach allows the system to capture a more comprehensive understanding of the bug report’s textual content.
Refining Recommendations with Interaction History
What truly sets TriagerX apart is its innovative “interaction-based ranking” methodology. After generating an initial content-based ranking, TriagerX refines it by considering developers’ past involvement with similar, already-fixed bugs. This includes analyzing commits, pull requests, discussions, and assignments. The system even accounts for the recency of these interactions, giving more weight to recent contributions, much like a human triager would.
How TriagerX Works: A Closer Look
The TriagerX framework is composed of three main components:
- Content-based Ranker (CBR): This component uses an ensemble of smaller PLMs (like RoBERTa and DeBERTa) to extract rich semantic information from bug reports. It combines features from multiple layers of these PLMs and uses a specialized CNN-based classifier to make initial developer predictions. This design allows it to achieve high accuracy while being more efficient than larger, single PLM models.
- Interaction-based Ranker (IBR): The IBR identifies similar bugs using a powerful text similarity model (Siamese BERT networks). It then calculates an “interaction score” for each developer based on their historical contributions to these similar bugs. Different types of interactions (e.g., a code commit versus a discussion comment) are weighted differently, and an exponential decay function ensures that recent activities have a stronger influence.
- Rank Aggregator (RAgg): Finally, the RAgg component intelligently combines the scores from both the CBR and IBR. It uses a Weighted Ranking Aggregation method, which can be tuned to give appropriate importance to content-based or interaction-based insights depending on the specific software repository. This ensures that the final recommendation is a balanced and accurate reflection of both the bug’s content and the developers’ historical expertise.
Also Read:
- Navigating the Landscape of Automated Code Review: A Comprehensive Analysis
- Unveiling AI’s Flaws: A New Framework for Classifying Defects in AI Software
Impressive Performance and Real-World Impact
Across five different datasets, TriagerX consistently outperformed nine other transformer-based methods, including leading state-of-the-art baselines. It often improved Top-1 and Top-3 developer recommendation accuracy by over 10%. For instance, on the OpenJ9 dataset, TriagerX achieved a 54% higher Top-1 accuracy than the LBT-P baseline and 58% higher than the closest large PLM-based baseline.
The framework has been successfully deployed in the development environment of a large industry partner, IBM. Here, TriagerX not only provides developer recommendations but also suggests component assignments, which act as proxies for team assignments—a particularly useful feature during developer turnover or team changes. For component recommendations, it outperformed baselines by up to 10%, and for developer recommendations, by an impressive 54%.
The industrial deployment has shown promising results, with TriagerX operating efficiently on CPU-only machines and providing real-time recommendations. Initial feedback from developers has been encouraging, highlighting its value as a starting point for triaging complex bugs. The team is also exploring future enhancements, such as providing contextual explanations for its recommendations to further build user confidence.
This innovative work demonstrates a significant leap forward in automated bug triaging, offering a more accurate, robust, and context-aware solution for software development teams. You can read the full research paper for more technical details at arXiv.org.
