TLDR: A new research paper introduces a ‘goal identification’ methodology to evaluate explainable reinforcement learning (XRL) algorithms. By training Ms. Pacman agents with distinct goals and providing explanations, the study tested if 100 users could accurately predict the agent’s objective. Findings show that explanations focusing on future outcomes (like Dataset Similarity Explanation) performed best, while those explaining only the next action struggled. Crucially, user confidence and self-reported understanding often did not correlate with actual accuracy, revealing a significant issue of user overconfidence in interpreting AI explanations.
Understanding how artificial intelligence (AI) makes decisions is crucial, especially in complex systems like reinforcement learning (RL). Explainable Reinforcement Learning (XRL) aims to make these decisions transparent, but evaluating the effectiveness of these explanations, particularly in comparative settings, has been a challenge. A recent research paper, “A Comparative User Evaluation of XRL Explanations using Goal Identification,” introduces a novel approach to tackle this problem by focusing on whether users can identify an AI agent’s underlying goal from its explanations.
The core idea behind this new evaluation method, termed ‘goal identification,’ is to test if users can accurately determine the reward function an agent is trying to maximize over time. Unlike previous methods that often focused on predicting an agent’s next action, this approach delves into the agent’s long-term objectives. The researchers trained multiple agents within the classic Atari Ms. Pacman environment, each with a unique reward function leading to different goals. For instance, one agent might aim to ‘Eat Dots,’ another to ‘Eat Energy Pills and Ghosts,’ a third to ‘Survive,’ and a fourth to ‘Lose a Life.’ These goals were designed to be strategically distinct and range from intuitive to counterintuitive for human participants.
The study involved 100 participants who were presented with an agent’s decision-making explanation and then asked to predict which of the four goals the agent was pursuing. Four different XRL explanation mechanisms were evaluated: Dataset Similarity Explanation (DSE), TRD Summarisation, Specific and Relevant Feature Attribution (SARFA), and Optimal Action Description (OAD). These mechanisms varied in what they explained (future outcomes versus next actions) and their medium (video, text, saliency maps).
Key Findings on Explanation Accuracy
The results revealed significant differences in how well users could identify agent goals based on the explanation provided. The Dataset Similarity Explanation (DSE) achieved the highest accuracy at 53.0%, followed by TRD Summarisation at 34.9%. Notably, DSE and TRD Summarisation were the two mechanisms designed to explain an agent’s future outcomes. In contrast, Optimal Action Description (OAD) and Specific and Relevant Feature Attribution (SARFA), which focused on explaining the agent’s next action, had much lower accuracies of 28.7% and 22.5%, respectively. For context, random guessing would yield an expected accuracy of 25%.
The study also found that the specific goal being explained influenced the accuracy of some mechanisms. For example, TRD Summarisation performed significantly better when explaining the ‘Lose a Life’ goal, likely because it was the only goal with negative rewards, making it easier for users to distinguish. DSE struggled with goals that had strategic overlap, such as ‘Eat Dots’ being frequently mistaken for ‘Eat Energy Pills and Ghosts’. SARFA consistently led users to misinterpret the agent’s decision-making, often predicting a different goal than the actual one.
The Disconnect Between Confidence and Accuracy
Perhaps one of the most striking findings was the weak correlation between users’ self-reported confidence, ease of identification, and understanding, and their actual accuracy. Users were often highly overconfident in their predictions. For DSE, there was a positive correlation between confidence and accuracy, meaning more confident users were generally more accurate. However, for OAD, the opposite was observed: accuracy decreased as confidence increased. For TRD Summarisation and SARFA, confidence showed little to no correlation with accuracy.
This highlights a critical challenge in XRL: self-reported metrics, while common, may not reliably indicate an explanation’s true effectiveness. Users might feel they understand an explanation or are confident in their choice, but their objective performance tells a different story. This overconfidence could have serious implications in real-world debugging scenarios where misunderstanding an AI’s intent could lead to significant errors.
Also Read:
- Building AI That Welcomes Change: A New Approach to Corrigible Goals
- Unlocking Trust: The Power of Interactive AI Explanations
Implications for Future XRL Research
The paper concludes that there are substantial performance gaps in current XRL literature, with only one of the tested mechanisms (DSE) achieving significantly above-random accuracy for all goals. The findings underscore the importance of objective evaluation methodologies over subjective user feedback alone. The authors suggest improvements for future user studies, such as refining explanation descriptions, increasing the number of questions, and incorporating attention checks to ensure user engagement.
This research provides a valuable framework for evaluating XRL explanations and sheds light on the complexities of human-AI interaction in understanding agent goals. For more details, you can read the full research paper here.
