TLDR: A new research paper introduces a framework to define “informative” explanations for AI models. It reveals that popular local post-hoc explanation algorithms (like SHAP, LIME, gradient explanations, and anchors) are often not informative for complex, black-box AI models. True informativeness is achieved only when models are simpler or when explanations are enriched with additional information, posing significant challenges for AI auditing and regulation.
In the rapidly evolving world of artificial intelligence, machine learning models are increasingly making decisions that profoundly impact people’s lives. From loan applications to job interviews, automated systems are at the forefront, leading to a growing demand for transparency and accountability. This demand has spurred the field of Explainable AI (XAI), with many researchers suggesting that local post-hoc explanation algorithms can offer valuable insights into these complex models.
However, a recent research paper titled “Informative Post-Hoc Explanations Only Exist for Simple Functions” by Eric Günther, Balázs Szabados, Robi Bhattacharjee, Sebastian Bordt, and Ulrike von Luxburg, challenges this widely held belief. Published on August 18, 2025, this paper introduces a rigorous, learning-theory-based framework to define what it truly means for an explanation to provide meaningful information about a decision function.
What Makes an Explanation “Informative”?
The core of this research lies in its novel definition of an “informative explanation.” The authors propose that an explanation is informative if it helps to reduce the complexity of the space of plausible decision functions. Imagine you have a set of possible AI models that could have made a particular decision. An informative explanation, according to this framework, should significantly narrow down that set, giving you a clearer picture of the underlying model’s behavior.
The paper uses a concept from learning theory called “Rademacher complexity” to quantify this reduction in complexity. If knowing an explanation leads to a strictly smaller Rademacher complexity for the set of remaining plausible functions, then the explanation is considered informative.
The Surprising Truth: Complex Models, Uninformative Explanations
One of the paper’s most significant findings is that many popular explanation algorithms are often *not* informative when applied to complex decision functions. This includes widely used methods such as:
- Gradient explanations (like LIME, SmoothGrad, integrated gradients)
- SHAP (Shapley Additive Explanations)
- Anchor explanations
- Counterfactual explanations (specifically, “weak” counterfactuals)
For instance, the research shows that gradient explanations and counterfactual explanations are non-informative for the space of all differentiable functions. Similarly, SHAP and anchor explanations are not informative for decision trees with arbitrary depth. This implies that for the very “black-box” models where explanations are most needed—such as deep neural networks or large random forests—these local post-hoc explanations may not provide any tangible information about the model’s overall behavior.
The authors argue that this isn’t necessarily a flaw in the algorithms themselves, but rather a fundamental mathematical limitation: trying to explain arbitrarily complex decision functions with pointwise explanations at individual data points is inherently difficult.
When Do Explanations Become Informative?
The paper doesn’t just present a negative outlook; it also identifies conditions under which explanations *can* become informative. These conditions often require stronger assumptions about the decision function or an enrichment of the explanation itself:
- **Simpler Functions:** Explanations become informative when applied to simpler function classes. For example, gradient explanations are informative for functions with bounded and Lipschitz continuous gradients (meaning their changes are smooth and predictable). SHAP and anchor explanations become informative for decision trees with limited depth.
- **Enriched Explanations:** Instead of simplifying the model, explanations can be made informative by providing additional information. For instance, if gradient explanations are guaranteed to be “locally stable” (meaning they don’t change wildly in a small neighborhood around the data point), they become informative even for functions with bounded gradients. Similarly, “perfect anchors” (those with 100% precision) and “strong counterfactuals” (which guarantee a region of consistent behavior) are informative because they reveal information about a whole region, not just a single point.
Also Read:
- Holistic Explainable AI: A New Framework for End-to-End AI Transparency
- Unlocking Graph Neural Network Decisions with Class-Level Explanations
Implications for AI Auditing and Regulation
The findings have profound implications, particularly for high-risk applications of AI and regulatory frameworks like Article 86 of the AI Act in the European Union. The paper suggests that unless an AI system’s developer explicitly states that the decision function is “simple” or that an explanation is locally stable, current local post-hoc explanations might be meaningless. They may not provide the kind of verifiable, contestable information that legislators likely envisioned.
This research highlights a critical gap: while explanations are often sought to understand complex models, their ability to truly inform us about the model’s underlying logic is limited by the model’s complexity. Future work will need to explore how “enriched” explanations can be made suitable for auditing and regulation, or if alternative approaches are necessary.
For more in-depth details, you can read the full research paper here.
