TLDR: This paper introduces a method to enhance AI explainability by embedding Large Language Models (LLMs) within standardized analytical frameworks like Question–Option–Criteria, Sensitivity Analysis, and Game Theory. It proposes a layered architecture that separates the LLM’s opaque reasoning from transparent, auditable process steps. Empirical evaluations demonstrate that this approach can reproduce human-level decision logic in various contexts, leading to more reliable, interpretable, and verifiable AI-supported decision-making.
Artificial Intelligence (AI) has made incredible strides, but many advanced systems, especially those powered by deep learning and large language models (LLMs), often operate as “black boxes.” This means their internal decision-making processes are difficult for humans to understand, leading to concerns about accountability, reliability, and ethical deployment, particularly in critical sectors like finance and healthcare.
Traditional Explainable AI (XAI) methods have tried to shed light on these black boxes, often by explaining what factors influenced a model’s decision. However, these methods frequently fall short in explaining *how* the reasoning process unfolded and can be overly technical for non-experts. A new research paper, “Increasing AI Explainability by LLM Driven Standard Processes,” by Marc Jansen and Marcel Pehlke, proposes a fresh perspective: instead of just explaining opaque models after the fact, AI systems can be designed to reason through processes that are inherently explainable.
Embedding LLMs in Transparent Processes
The core idea is to integrate Large Language Models (LLMs) into well-established analytical frameworks that are transparent by design. These “standard processes” include methods like Question–Option–Criteria (QOC) analysis, Sensitivity Analysis, Game Theory, and Risk Management. These frameworks have long been used in human decision-making and systems analysis because they have clear structures, traceable reasoning steps, and outputs that are easy to interpret. By embedding LLM reasoning within these formalized structures, the approach transforms what might otherwise be opaque AI inference into transparent, auditable decision trails.
The paper introduces a layered architecture to achieve this. It conceptually separates the “unexplainable reasoning space” of the LLM from the “explainable process space” above it. This boundary is called the “Explainability Barrier.” Below this barrier, the LLM performs its complex, internal computations. Above it, the standardized analytical processes operate, providing measurable, interpretable, and accountable outputs. The goal is to systematically move more decision logic above this barrier, maximizing traceability.
How Standard Processes Leverage LLMs
-
Question–Option–Criteria (QOC): This framework helps evaluate decision alternatives against weighted criteria. In an LLM-driven setup, AI agents can identify options and criteria, evaluate them, and consolidate findings, providing a clear rationale for the chosen option.
-
Sensitivity Analysis: Based on models like the Vester model, this process quantifies how different variables influence each other within a system. LLM agents can define system variables, estimate their direct influences, and help classify their roles (e.g., active drivers or sensitive factors), making systemic dynamics transparent.
-
Game Theory Models: Whether it’s a normal-form game (simultaneous decisions) or a sequential game (ordered decisions), LLMs can help identify players, strategies, and payoffs. This allows the system to compute stable outcomes like Nash equilibria or Pareto-optimal paths, explaining strategic behavior.
-
Risk Management: LLM agents, representing different stakeholders, can identify, consolidate, and evaluate potential project risks based on probability and impact. This helps in prioritizing risks and understanding stakeholder perspectives.
In each case, the LLM’s role is to perform specific sub-tasks like generating inputs or estimating relationships, while the overall process logic remains structured and analyzable. This hybrid approach combines the linguistic and contextual strengths of LLMs with the analytical rigor of established decision-support frameworks.
Empirical Validation and Promising Results
The researchers evaluated this architecture through several prototype implementations:
-
QOC Evaluation: Applied to real-world investment decisions from a Decentralized Autonomous Organization (DAO), the LLM-driven QOC framework showed that larger, more capable LLMs (like GPT-5) tended to make more conservative and risk-aware decisions, often aligning with human-level decision logic while minimizing costly errors.
-
Sensitivity Analysis Evaluation: The multi-agent LLM pipeline successfully replicated the structure and reasoning of a Vester-style sensitivity analysis for logistics, demonstrating human-level coherence in identifying factors, mapping causal links, and interpreting systemic roles.
-
Game Theory Evaluation: A role-conditioned, LLM-driven extensive-form game was tested against an abstraction of the Cuban Missile Crisis. The system reliably reproduced the end-state logic, with a high percentage of paths ending in “De-escalate,” mirroring the patterned reasoning dynamics of sequential games.
These evaluations suggest that LLMs, when embedded within formalized decision processes, can act as “explainability enablers,” producing powerful yet transparent AI-supported decision-making.
Also Read:
- Unpacking AI Decisions: How LLM Agents Create Transparent Reasoning Paths
- Unpacking LLM Decisions: New Insights on Data Influence and Layer Selection
Looking Ahead: Challenges and Future Directions
While the results are promising, the paper also acknowledges limitations. These include the potential for retrieval-augmented generation (RAG) pipelines to propagate outdated or biased information, the challenge of “hallucination propagation” where agents generate fabricated data, and the risk of misaligned outputs in multi-agent settings. The inherent opacity of LLMs at a cognitive level also means that full transparency is not guaranteed, only structured traces above the explainability barrier.
Future work aims to address these issues through technical refinements, such as integrating source-credibility filters in RAG pipelines, formalizing process orchestration, and developing “process-aware LLMs” that have structured reasoning patterns built into their intrinsic latent space. The ultimate goal is to create AI systems that are not only explainable in their outputs but also explainable by design in their reasoning.
