TLDR: This research paper addresses the limitations of using only textual similarity for recovering traceability links between natural language requirements and programming language code. Through a large-scale empirical evaluation, the study confirms that textual similarity performs poorly in these NL-PL tasks due to a semantic gap. To overcome this, the authors propose an approach that integrates multiple domain-specific auxiliary strategies, such as code dependencies, user feedback, and fine-grained semantic analysis, into two models: a Heterogeneous Graph Transformer (HGT) for supervised learning and a prompt-based Gemini 2.5 Pro for unsupervised learning. Experimental results show that both multi-strategy models significantly outperform existing state-of-the-art methods, demonstrating the effectiveness of combining diverse information sources to achieve more accurate and robust traceability link recovery. The study also provides insights into the applicability of these models based on project scale and resource availability.
Software traceability is a crucial aspect of software development, allowing teams to track and understand the connections between different components throughout a project’s lifecycle. Imagine being able to link a customer’s initial request directly to the specific lines of code that implement it, or to the test cases that verify it. This ability is vital for tasks like analyzing the impact of changes, ensuring safety, and verifying coverage.
However, a significant challenge arises when trying to establish these links between natural language (NL) artifacts, such as requirements documents, and programming language (PL) artifacts, like source code. This is known as Natural Language–Programming Language (NL-PL) Software Traceability Link Recovery (TLR). For a long time, the primary method for finding these links has been textual similarity – essentially, looking for shared words or phrases between documents. But a recent study highlights that this approach is often insufficient due to a fundamental “semantic gap” between how humans describe things in natural language and how computers implement them in code.
The Limitations of Textual Similarity
Researchers conducted a large-scale evaluation across various types of traceability tasks. They found that while textual similarity works reasonably well for linking artifacts within the same language type (e.g., natural language to natural language, or programming language to programming language), its effectiveness drops significantly when trying to link natural language to programming language. This is because programming code is often abstract, uses specific keywords and API calls, and may not directly describe its functional semantics in a way that aligns with a natural language requirement. A single requirement might also correspond to multiple, modular code segments, some of which are only briefly mentioned in the requirement, leading to low textual overlap.
To quantify this, the study introduced a metric called the “Difference Ratio,” which measures how well textual similarity can distinguish between true and false links. Projects involving NL-PL artifacts, such as requirements-to-code, consistently showed a lower Difference Ratio, confirming the inherent difficulty of these tasks using text alone. This empirical evidence strongly suggests that relying solely on textual similarity is not enough for robust NL-PL traceability.
A Multi-Strategy Approach to Bridging the Gap
To overcome these limitations, the researchers proposed an innovative approach that integrates multiple domain-specific auxiliary strategies. These strategies, identified through careful empirical analysis, provide richer contextual information beyond just text. The approach was implemented and evaluated using two powerful models: the Heterogeneous Graph Transformer (HGT) for supervised learning scenarios (where some labeled data is available) and the prompt-based Gemini 2.5 Pro for unsupervised settings (where no labeled data is needed for training).
Key strategies integrated into these models include:
- Code Dependency: This strategy leverages the structural relationships within code, such as one code file importing another, or one class extending another. If a requirement is linked to one code artifact, other code artifacts that are structurally dependent on it might also be relevant.
- User Feedback: In real-world scenarios, developers often have a small set of high-confidence, manually confirmed links. This strategy incorporates such user feedback as a guiding signal to enhance the automated link recovery process.
- Fine-Grained Semantic Analysis: Instead of comparing entire code files, this strategy breaks down code into smaller, more specific components like class names, method names, and comments. By comparing these fine-grained elements with requirements, it can identify strong local semantic alignments even if the overall textual similarity is low.
The Heterogeneous Graph Transformer (HGT) model was designed to represent these diverse relationships as different types of “edges” in a graph, allowing it to learn from the complex interplay of textual and structural information. For the Gemini 2.5 Pro model, these strategies were embedded as additional input information within prompt templates, guiding the large language model to make more informed decisions about link existence.
Impressive Results and Future Directions
The experimental results were highly encouraging. Both the multi-strategy HGT and Gemini 2.5 Pro models significantly outperformed their original counterparts (without strategy integration) and current state-of-the-art methods. For instance, the multi-strategy HGT achieved an average F1-score improvement of 3.68% over HGNNLink, while Gemini 2.5 Pro showed an even more substantial 8.84% improvement across twelve open-source projects. This clearly demonstrates the effectiveness of integrating multiple strategies in enhancing overall model performance for the challenging requirements-to-code traceability task.
Interestingly, the study found that HGT-All (the multi-strategy HGT) tended to perform better on large-scale industrial projects, likely because these projects offer more complex structural relationships for the graph model to learn from. Conversely, Gemini-All (the multi-strategy Gemini) showed more pronounced improvements on smaller projects, where code semantics might be more directly aligned with requirements. This suggests that the choice of model might depend on project scale and available resources.
Also Read:
- Understanding Why Code Changes: A Large-Scale Study with AI
- Understanding AI Model Reuse in Open-Source Software
Implications for the Software Industry
For researchers, this study emphasizes the importance of integrating new strategies with existing ones to achieve synergistic effects. It also highlights that while large language models like Gemini are powerful, their performance in unsupervised settings is highly sensitive to the quality and relevance of the auxiliary information provided. HGT-based approaches, while robust, require labeled data and are sensitive to the scale of the dataset.
For practitioners, the findings offer a path to more accurate traceability. However, it comes with considerations: Gemini-All, while effective, can incur high computational and financial costs due to API usage, making it less suitable for real-time or budget-constrained scenarios. HGT-All requires high-quality labeled training data, demanding significant human resource investment for annotation. The researchers suggest that enterprises could adapt these approaches by fine-tuning internal heterogeneous graph neural networks or large language models to better suit their specific project needs, offering a flexible and adaptable paradigm for future advancements in software traceability. You can read the full research paper here: Natural Language–Programming Language Software Traceability Link Recovery Needs More than Textual Similarity.
