TLDR: This research explores how visual analytics (VA) can make artificial intelligence (AI) systems more understandable and trustworthy. It addresses the “black box” problem of AI by combining AI models with interactive visualizations. The work defines trust in AI, categorizes existing VA solutions, proposes workflows for various stages of the AI pipeline (data processing, feature engineering, hyperparameter tuning, model understanding, debugging, and comparison), and presents several VA dashboards developed by the author. It highlights the importance of VA in high-stakes domains like healthcare and identifies future research directions to enhance AI transparency and reliability.
Our society is increasingly reliant on artificial intelligence (AI) systems for solving complex problems, from recommending movies to assisting in critical medical diagnoses. While AI promises significant advancements, such as potentially reducing medical misdiagnoses and their associated economic burden, a major hurdle to its widespread adoption is the lack of transparency. Many AI systems operate as “black boxes,” providing predictions without revealing how they arrived at those conclusions. This opacity can make it difficult for experts to trust and depend on AI.
Visual analytics (VA) offers a powerful solution to this challenge by integrating AI models with interactive visualizations. These specialized charts and graphs empower users to apply their domain expertise, helping to refine and improve AI models and bridge the gap between AI and human understanding. This research defines, categorizes, and explores how VA solutions can build trust across the various stages of a typical AI development process.
Understanding the AI Pipeline Through Visuals
The research proposes a design space for innovative visualizations and provides an overview of several VA dashboards developed to support critical tasks within the AI pipeline. These tasks include data processing, feature engineering, hyperparameter tuning, understanding, debugging, refining, and comparing models. The work highlights that while AI research initially focused on predictive performance, there’s a renewed emphasis on data-centric AI, along with intrinsic algorithmic interpretability and post-hoc explainability, which are crucial for assessing an AI model’s overall knowledge.
The study identifies five levels of trust in AI, ranging from the reliability of raw data to the evaluation of subjective user expectations. It also categorizes 542 peer-reviewed papers on visualization techniques for enhancing trust in AI models, providing a comprehensive overview of the field.
Key Visual Analytics Tools and Workflows
The research introduces several VA dashboards, each designed to address specific challenges at different stages of the AI pipeline:
-
HardVis: This dashboard tackles the problem of imbalanced data by using highly configurable undersampling and oversampling methods. It provides multiple views to help users determine the optimal data distribution, remove problematic samples, and interactively oversample others, making the process transparent.
-
FeatureEnVi: Designed to support feature engineering, this tool assists users in selecting, transforming, and creating new features. By examining the impact of features using various statistical metrics, users can enhance predictive performance and reduce computational costs.
-
VisEvol: This dashboard facilitates hyperparameter search using evolutionary optimization. It enables users to create new hyperparameter sets and preserve robust ones within AI ensembles, providing clarity on how to select hyperparameters for single models or complex ensembles.
-
t-viSNE: For exploring t-SNE projections, a common method for visualizing high-dimensional data, this dashboard helps users evaluate projection quality and understand the algorithm’s decision-making process when forming clusters.
-
DeforestVis: This tool employs decision stumps to create simpler, more explainable surrogate models that approximate the behavior of complex AI models. Users can balance complexity and fidelity, adjust individual rules, and reason through specific test cases.
-
VisRuler: Focused on model debugging, this dashboard allows users to examine rules derived from ensemble methods like bagged and boosted decision trees. It supports selecting high-performance models, analyzing feature contributions, and facilitating case-based reasoning.
-
StackGenVis: This dashboard is designed for aligning data, algorithms, and models in stacking ensemble learning. It helps users create effective stacking ensembles from scratch by exploring various perspectives and monitoring the training process.
Also Read:
- Navigating Explainable AI: A Framework for Understanding Research Contradictions
- Keeping AI Models Reliable: A New Approach to Monitoring Performance During Adaptation
Impact and Future Directions
The insights from this research have significant implications, especially with regulations like GDPR and the AI Act requiring clear explanations for automated decisions. In high-stakes fields like healthcare and finance, the ability to explain AI’s reasoning is becoming ethically and legally crucial. The methods described are applicable to both academic research and business intelligence, fostering greater explainability and trust in decision-making processes where human lives or significant financial outcomes are at stake.
The research also identifies several future opportunities for visualization researchers, practitioners, and AI experts. These include improving popular explainable AI methods, exploring new neural network approaches, advancing confirmatory visual analytics, quantifying input and output uncertainty, developing more rigorous evaluation and benchmarking methods, enhancing model deployment, and addressing underexplored areas in visual analytics suchates boosting and stacking ensemble learning methods, multi-label and regression problems, and reinforcement learning.
For more detailed information, you can refer to the full research paper available at arXiv:2507.10240.
