TLDR: A new approach uses generative AI and a system’s process data (inputs, outputs, documentation) to create natural-language explanations for debugging distributed software systems. This “post-hoc” method, demonstrated on a Java system, leverages Large Language Models (LLMs) to summarize complex interactions, making debugging more efficient and understandable for developers, even for unfamiliar systems.
Debugging software, especially complex and distributed systems, remains a significant challenge for developers. Traditional manual debugging methods are often time-consuming and resource-intensive, particularly when dealing with systems composed of many interacting components like web services. As software complexity grows, the need for more efficient debugging tools becomes critical.
Researchers Dennis Schiese and Andreas Both from Leipzig University of Applied Sciences have introduced an innovative approach to address this problem. Their work, detailed in the paper “Post-hoc LLM-Supported Debugging of Distributed Processes”, proposes using a system’s actual process data in conjunction with generative Artificial Intelligence (AI) to create natural-language explanations. These explanations are designed to guide developers more efficiently in understanding system behavior and identifying potential errors.
The core idea is to leverage runtime process data, interface information, and documentation to generate clear, human-readable explanations. This approach can be applied to any system, regardless of its scale or programming language, making it highly versatile. It aims to provide a good understanding of a process, even if developers are not familiar with every detail of the system.
How the Approach Works
The method focuses on “post-hoc” analysis, meaning it inspects the system after it has run. For each method call within a component, relevant data such as input, output, docstrings, and caller information are logged. This data is then stored, for example, in a triplestore for persistent access.
A key innovation is the use of Large Language Models (LLMs) to process this data. Instead of developers sifting through thousands of method calls, LLMs extract the most relevant aspects from sub-called methods’ explanations and integrate them into a concise, natural-language explanation for the current method. This hierarchical aggregation ensures developers receive a compact overview of what happened within a specific process, propagating relevant information while filtering out less important details.
The researchers implemented a demonstrator using a component-based Java system, specifically Qanary, which is well-suited for analyzing distributed systems. The demonstrator is an open-source web application that allows users to select an executed process, a component, and a starting method. It then generates explanations for the selected method and its subordinate calls. Users can choose between template-based generation or LLM-based generation, and even decide whether to include a method’s docstring in the explanation.
Also Read:
- Large Language Models Struggle to Accurately Verify Code, Study Finds
- Making AI Transparent: LLM-Powered Explanations for Knowledge Graph QA
Benefits and Future Directions
This LLM-driven debugging approach offers several significant benefits. It streamlines the debugging process by providing clear, aggregated explanations, reducing the manual effort and time required. Its language-agnostic nature makes it broadly applicable across different software environments. The demonstrator proves the feasibility of using process-dependent data, like input arguments and return values, to generate useful explanations.
Looking ahead, the researchers suggest further improvements by incorporating additional data sources such as source code or performance information. This could lead to even higher quality explanations, further enhancing the efficiency of manual debugging in increasingly complex software landscapes.
