TLDR: A new research paper proposes a framework to categorize Explainable AI (XAI) studies, addressing contradictions and a lack of concrete design recommendations in the field. The framework uses three dimensions—’Who’ (user expertise), ‘Why’ (analysis purpose and decision-making type), and ‘What’ (data type and task characteristics)—to analyze studies. By applying this framework, the authors explain conflicting findings regarding automation bias, the utility of global explanations, and the level of explainability needed in conversational AI systems. The paper also provides guidelines for future XAI research to improve comparability and generalizability of findings.
Artificial intelligence (AI) is increasingly assisting people with complex tasks like data analysis. However, the field of Explainable AI (XAI), which focuses on making AI systems understandable, faces challenges due to conflicting research findings and a lack of clear design guidelines. A recent survey by Hamzah Ziadeh and Hendrik Knoche from Aalborg University addresses these issues by proposing a new method for categorizing and comparing XAI studies.
The researchers identified three main problems in current XAI research: inadequate descriptions of tasks, studies conducted without real-world context, and insufficient testing with the actual target users. To tackle these, they suggest that studies should clearly report on their users’ expertise in their domain, AI, and data analysis to show how broadly applicable their findings are. They also propose guidelines for designing and reporting XAI tasks, aiming to help the XAI community better navigate this rapidly expanding field.
A New Framework for Understanding XAI Studies
The core of this research is a framework that categorizes XAI studies along three dimensions: ‘Who,’ ‘Why,’ and ‘What.’ This helps to distinguish tasks between different studies and understand why results might vary.
Who is analysing?
This dimension focuses on the user’s knowledge. Users can have different levels of expertise in their specific domain (e.g., a doctor in medicine), data analysis methods, and AI models. For instance, domain experts might rely less on detailed local explanations if they can interpret raw data, preferring global explanations that show how the AI model works overall. Conversely, those new to a domain might need more basic explanations. The survey highlights a significant gap: many studies do not involve actual domain experts, leading to designs that might not meet their real-world needs.
Why are they analysing?
This dimension looks at the purpose of the analysis and the type of decision-making involved. Tasks are categorized into diagnostic (clear objectives, verifiable conclusions about the past/present), prognostic (making predictions about the future with uncertainty), and inferential (explaining past/present phenomena with incomplete data). Decision-making can be exploratory (discovering new insights without a prior hypothesis) or confirmatory (testing a pre-existing hypothesis). The paper notes that many XAI studies focus on exploratory analysis, which might not reflect how domain experts, who often start with a hypothesis, use AI systems in real life.
What are they analysing?
This dimension considers the nature of the data and the task’s characteristics. Data can be data-driven (structured datasets with variables) or information-driven (text, images, videos). The type of data influences the kind of AI assistance needed; for example, information-driven tasks might rely on AI highlighting points of interest. Other factors include whether the data is quantitative or qualitative, the type of quantitative data (categorical or ordered), and the complexity (number of variables) and severity (impact of the decision) of the task. A key finding is that many studies fail to report these details, making comparisons difficult.
Also Read:
- Unpacking AI Decisions: A New Logic for Contrastive Explanations
- Bridging the Gap: Advancing Creative Abduction in AI Systems
Reconciling Contradictory Results
The proposed categorization method helps explain why different studies yield conflicting results. For example:
-
Automation Bias: This refers to users over-relying on AI recommendations. The survey suggests that automation bias is more prevalent in studies where participants have no personal stake or knowledge in the task (e.g., crowd workers) or when tasks are prognostic and inherently more difficult. Studies with real domain experts performing relevant tasks showed much weaker or no evidence of automation bias.
-
Global Explanations: These explain how the AI model works generally. Some studies found them helpful, increasing trust, while others found they increased workload without improving performance. The paper suggests that global explanations are more valuable in data-driven prognostic tasks where understanding the AI’s pipeline is crucial. In information-driven tasks, users might not need global explanations as they can verify information by reviewing the source data themselves.
-
Explainability in Conversational AI (CAI): There’s disagreement on whether CAI systems should explain their analytical methods. The survey indicates that analytical and domain experts often desire such explanations or control over the process, especially in prognostic tasks. In contrast, laypeople or crowd workers in non-data analysis tasks may not require detailed methodological explanations, preferring simpler evidence for recommendations.
This research provides valuable insights for the XAI community, offering a structured way to analyze and compare studies. By better defining users, tasks, and data, researchers and designers can create more effective and context-appropriate XAI systems. For more details, you can read the full paper here.
