TLDR: This research introduces a modular LLM-agent pipeline that fosters Explainable AI by externalizing reasoning into auditable artifacts. It integrates frameworks like Vester’s Sensitivity Model and game theory, allowing LLMs to mimic expert decision-making with transparent, inspectable steps, moving beyond opaque ‘black-box’ outputs. Evaluation on a logistics case showed high alignment with human baselines and strong methodological quality, demonstrating the system’s ability to reliably follow structured analytical processes.
Artificial intelligence, particularly Large Language Models (LLMs), has made incredible strides, offering solutions to complex problems. However, a significant challenge remains: understanding how these models arrive at their conclusions. This “black-box” nature can undermine trust, especially in critical decision-making scenarios. A new research paper, “LLM Driven Processes to Foster Explainable AI,” by Marcel Pehlke and Marc Jansen from the University of Applied Sciences Ruhr West, addresses this by proposing a novel approach to make AI reasoning transparent and auditable.
The core idea is to move beyond opaque, one-shot answers from LLMs. Instead of just getting a final decision, this system externalizes the reasoning process into clear, inspectable steps. Imagine an AI that not only gives you an answer but also shows you its thought process, much like a human expert would explain their logic.
Structured Reasoning with LLM Agents
The researchers developed a modular pipeline of LLM agents designed for decision support. This system integrates three well-established frameworks, allowing LLMs to mimic expert workflows with transparent steps:
- Vester’s Sensitivity Model: This framework helps analyze complex, interdependent problems by identifying key drivers, sensitivities, and feedback loops within a system. It involves creating an “impact matrix” to score how different variables influence each other and then classifying variables into roles like “active,” “critical,” “reactive,” or “buffering.”
- Normal-Form Games: Used for analyzing strategic interactions where multiple agents make simultaneous choices. It helps identify optimal strategies and outcomes, such as Nash equilibria, where no player can improve their situation by changing their strategy alone.
- Sequential Games: This framework models interactions where players take turns, represented as a decision tree. It uses concepts like “backward induction” to find credible strategies at each step, ensuring decisions are sound throughout the game.
What makes this approach unique is that LLM components (defaulting to GPT-5 in their tests) are paired with deterministic analyzers. This means that instead of just getting a final, unexplainable output, the system generates traceable intermediate artifacts like influence matrices, payoff tables, and decision trees. These artifacts serve as an auditable record of the AI’s reasoning.
How It Works in Practice
The system is designed with modularity in mind. Each analytical framework is a self-contained process. For instance, in a normal-form game, modules identify players, objectives, and strategies, then an LLM assigns payoffs, and a deterministic analyzer computes equilibria. Finally, an interpretation agent summarizes the findings in natural language, even suggesting practical recommendations.
The Vester Sensitivity Analysis pipeline, for example, starts by transforming a natural-language scenario into a structured system definition, identifying variables and stakeholders. An LLM agent, mimicking a “stakeholder roundtable,” then scores the influence between each pair of variables, creating a signed, weighted impact matrix. This matrix is then used to classify variables into their systemic roles and identify feedback loops. The final step involves an interpretation agent that converts this detailed analysis into actionable guidance, explaining sensitivities, proposing intervention priorities, and recommending the best levers for change.
Real-World Application and Evaluation
To test their system, Pehlke and Jansen applied the Vester pipeline to a real-world logistics case study involving apparel transport from China to the Nordics. They conducted 100 independent runs and compared the LLM-generated results against a human-conducted study from the IVL Swedish Environmental Research Institute.
The evaluation showed promising results:
- Factor Alignment: The LLM pipeline consistently identified key external transport levers, with a mean factor alignment of 55.5% across all 26 factors, rising to 62.9% for the core transport-system factors. This indicates the agents “thought” along similar systemic lines as human experts.
- Role Consistency: While moderate (around 57% over matched factors), the role agreement was consistent with the inherent sensitivity of Vester’s method, where small changes in influence judgments can legitimately shift variable classifications.
- Methodological Quality: An LLM judge, using an eight-criterion rubric, scored the runs. The LLM-generated analyses consistently produced high-quality studies, with total scores tightly clustered between 77 and 98 (mean 92.97) against a human baseline of 93. This demonstrates that the pipeline reliably followed the form and reasoning steps of a Vester-style study, rather than just imitating text.
The researchers also applied their framework to the Canada housing crisis scenario, providing a problem statement about undersupply, rising costs, and other challenges. The Vester analysis pipeline identified salient levers and risks that closely aligned with those emphasized in the federal initiative “Build Canada Homes,” launched in September 2025. This was particularly notable because the LLM had no prior knowledge of these specific policy measures, demonstrating its ability to surface actionable insights consistent with contemporary policy design.
Also Read:
- New AI System ED2D Uses Evidence-Based Debate to Combat Misinformation
- AI Agents Reveal How Mental Schemas Shape Misinformation Responses
The Future of Explainable AI
This research marks a significant step towards Explainable AI (XAI) by implementing traceable frameworks. It shows that multi-agent LLM pipelines can reproduce known methods with transparent steps, delivering human-aligned and inspectable decision support. The modular design also allows for future enhancements, such as integrating an “expert layer” that can draw from real-time web searches, domain-specific models, or curated expert feedback to further improve the robustness and fidelity of the analysis.
For more details, you can read the full research paper here.
