TLDR: A research paper evaluates how Large Language Models (LLMs) like Claude 3.5 Sonnet, GPT-4o, and o3-mini perform in linking software documentation to code. The study found LLMs significantly outperform traditional methods in identifying trace links, achieving high precision but varying recall. While LLMs show strong understanding of core relationships, their explanations often lack complete detail, and they can struggle with precise intermediate steps in multi-step trace chains. The research highlights the importance of task framing and suggests LLMs are powerful assistants for traceability, best used in a human-in-the-loop design.
Software development relies heavily on clear documentation that explains how code works. However, keeping this documentation in sync with constantly evolving code is a major challenge. This is where ‘traceability’ comes in – the ability to link documentation to its corresponding code. Traditionally, this has been a manual and often difficult process, leading to outdated or incomplete information.
A recent research paper, titled “Evaluating the Use of LLMs for Documentation to Code Traceability,” explores how Large Language Models (LLMs) like Claude 3.5 Sonnet, GPT-4o, and o3-mini can help automate this complex task. Authored by Ebube Alor, SayedHassan Khatoonabadi, and Emad Shihab from Concordia University, Canada, the study provides a comprehensive look into the capabilities and limitations of these advanced AI models in bridging the gap between human-readable documentation and machine-executable code.
The Challenge of Linking Documentation and Code
Imagine a large software project with thousands of lines of code and extensive documentation. As the code changes, the documentation needs to be updated to reflect those changes. Manual linking is time-consuming and prone to errors. Existing automated methods, often based on keyword matching or statistical analysis, frequently fall short because they struggle to understand the deeper meaning and relationships between natural language descriptions and programming code. This ‘semantic gap’ means they might miss connections or create incorrect ones.
How LLMs Were Put to the Test
The researchers set out to evaluate LLMs across three key areas: how accurately they can identify trace links, the quality of their explanations for these relationships, and their ability to reconstruct multi-step connections between documentation and code. To do this, they created two new datasets from open-source projects, Unity Catalog and Crawl4AI, ensuring these projects were developed after the LLMs’ training cutoff dates to guarantee a fair evaluation on unseen data.
They compared the LLMs against traditional methods like TF-IDF, BM25, and CodeBERT. The LLMs were given a piece of documentation and a list of possible code elements, then asked to identify the links. This ‘one-to-many’ approach allowed the LLMs to consider all potential connections for a given documentation segment simultaneously.
Key Findings: LLMs Show Promise, But With Nuances
The study found that LLMs significantly outperformed traditional baselines in identifying trace links. For instance, the best-performing LLM achieved F1-scores of 79.4% and 80.4% across the two datasets, which is a substantial improvement over baselines that scored between 36.2% and 69.3%. LLMs consistently showed high precision, meaning when they identified a link, it was usually correct. However, their recall (identifying all correct links) varied more.
When it came to explaining the relationships between documentation and code, LLMs demonstrated a strong grasp of the fundamental connection, with over 97% ‘relaxed accuracy’ (meaning they got the gist). However, their ‘strict accuracy’ (providing a complete and precise explanation) was lower, ranging from 42.9% to 71.1%. This indicates that while LLMs understand the core relationship, they often miss subtle but important details.
For multi-step connections, where documentation might link to a class, which then links to a method, and then to a specific constant, LLMs were good at identifying the start and end points of these chains. However, they sometimes struggled with the precise intermediate steps. Errors often involved adding extra unnecessary steps, missing crucial intermediate steps, or substituting one intermediate step for another.
Common Pitfalls and How to Address Them
The research also highlighted common errors made by LLMs. Many false positives (incorrect links) stemmed from the LLMs making assumptions based on naming conventions or overgeneralizing architectural patterns. For example, an LLM might assume an inheritance relationship between classes just because their names are similar, even if the code doesn’t support it. Another issue was ‘phantom links,’ where the LLM linked to code mentioned in the documentation but not actually present in the provided code set.
Interestingly, the study found that providing too much context to the LLM, such as the entire documentation file instead of just the relevant segment, could actually hurt performance. This led to ‘context boundary violations,’ where the LLM linked to code outside the specific section it was supposed to be analyzing.
Also Read:
- AI’s Role in Code Logging: A Deep Dive into GPT-4o Mini’s Performance in Machine Learning Projects
- The Hidden Flaw: How Large Language Models Handle Bad Code Instructions
Implications for Software Development
These findings suggest that LLMs can be powerful assistants for software developers. They can automate the initial discovery of trace links, significantly reducing manual effort. However, human oversight remains crucial, especially for critical documentation or complex relationships. Developers can use LLMs for a quick ‘first pass’ at traceability, then manually review and refine the suggestions. Tools integrating LLMs should be designed to facilitate this human-AI collaboration, allowing developers to easily correct or refine suggested links and explanations.
The research also emphasizes the importance of how information is presented to LLMs. Breaking down documentation into manageable, focused segments and using a ‘one-to-many’ matching strategy is more effective than trying to process everything at once. This ensures the LLM can concentrate its attention and avoid output limitations.
For more detailed information, you can read the full research paper here.
