TLDR: A study investigated how medical experts perceive attention-based explanations in AI systems classifying biomedical documents. It found that while the AI model was accurate, attention weights were generally not perceived as helpful for explanations, though their usefulness varied significantly with visualization. Users preferred intuitive formats like background color or text brightness over precise bar length visualizations, which also imposed a higher cognitive load. In contrast, the model’s predicted probability was consistently seen as highly useful, regardless of visualization. The research emphasizes that simpler, user-friendly visualizations are crucial for effective AI interpretability in medicine.
Artificial intelligence, particularly models known as Transformers, has become incredibly powerful in various fields, from understanding language to analyzing images. However, their complex inner workings often make it difficult to understand *why* they make certain predictions. This lack of transparency, known as the ‘interpretability problem,’ is a significant hurdle, especially in critical domains like medicine.
One proposed solution for interpretability in Transformers is the ‘attention mechanism.’ This mechanism assigns weights to different parts of the input (like words in a document), suggesting which parts the model focused on most when making a decision. The idea is that higher attention weights might indicate more relevant features for the model’s prediction, thereby offering an explanation.
In evidence-based medicine, where doctors need to quickly assess large amounts of research to make informed decisions, AI systems that can explain their reasoning would be invaluable. Imagine an AI helping categorize biomedical literature; an explanation of its decision could build trust and improve efficiency. However, there’s an ongoing debate about whether these attention weights truly provide meaningful or trustworthy explanations. Furthermore, little research has explored how different ways of *visualizing* attention might affect how useful users perceive them to be.
To address these questions, a recent study titled User Perception of Attention Visualizations: Effects on Interpretability Across Evidence-Based Medical Documents, conducted by a team including Andrés Carvallo, Denis Parra, Peter Brusilovsky, Hernan Valdivieso, Gabriel Rada, Ivania Donoso, and Vladimir Araujo, set out to investigate these very issues. They developed a system within the Epistemonikos platform, a widely used resource for evidence-based medicine, that classifies biomedical articles and generates visual explanations based on attention weights from a Transformer model (specifically, XLNet, which outperformed other models like BERT and BioBERT in their classification task).
The researchers conducted a user study involving five medical experts from various specialties. These experts classified 200 biomedical articles each, mimicking a real-world scenario where clinicians categorize literature by study design. The study tested three different ways of visualizing attention in text, alongside a control condition with no visualization:
Visualizing Attention
- Background Color: Words with higher attention weights had a darker background shade.
- Word Luminance: More important words appeared darker in text brightness.
- Bar Length: A horizontal bar beneath each word, with its length proportional to the word’s importance.
Participants were asked to rate the helpfulness of both the attention-highlighted words and the model’s predicted probability (or certainty) in classifying the articles.
Key Findings from the Study
The study yielded several interesting and somewhat counter-intuitive results. While the Transformer model (XLNet) accurately classified the documents, the attention weights themselves were generally *not* perceived as particularly helpful for explaining the predictions. Users rated their usefulness relatively low, peaking around 3.0 on a 5-point scale.
However, this perception varied significantly depending on how the attention was visualized. Contrary to established principles of visual effectiveness, which often favor precise encodings like bar length for conveying quantitative information, users preferred more intuitive formats such as text brightness (luminance) or background color. For instance, in Systematic Reviews and Broad Syntheses articles, background color and luminance were rated more helpful than the bar length visualization. The bar length encoding was consistently perceived as less useful across all article types.
In stark contrast to attention-based explanations, the model’s predicted probability (how certain the model was about its classification) was consistently rated as highly useful across all article types, with mean scores above 4.0. Importantly, the way attention was visualized did not affect the perceived usefulness of these predicted probabilities.
The study also used the NASA-TLX (Task Load Index) to assess cognitive workload. The background color visualization was associated with the lowest mental, physical, and temporal demand, indicating a lower overall cognitive load. While the no-visualization condition showed the highest perceived performance and lowest frustration (likely due to user familiarity with the standard interface), it also had higher mental and temporal demands than the background color. Bar length and word luminance, on the other hand, generally led to higher cognitive load, with bar length showing the highest frustration and effort.
Also Read:
- Bridging the Gap: Real-World Insights into Explainable NLP Adoption and Challenges
- Making AI Explanations Accessible: Addressing the Needs of Users with Vision Impairments
Conclusion
The research suggests that while Transformer models are accurate, the utility of attention weights as explanations for medical experts is debatable and heavily influenced by their visual presentation. Simpler, more intuitive visualizations like background color appear to be more effective and less cognitively demanding than precise but potentially overwhelming formats like bar length. The model’s predicted probability, however, consistently proved to be a highly valuable and easily understood piece of information for users, regardless of how other explanations were presented.
This study highlights the importance of user-centered design in explainable AI, especially in high-stakes domains like medicine. Future work will explore more interactive explanations and involve broader user groups and medical tasks.
