TLDR: SAInT is a new Python-based visual tool that simplifies understanding Machine Learning models for both AI researchers and domain experts. It integrates local and global sensitivity analysis (using LIME, SHAP, and eFAST) within an interactive graphical interface, enabling users to train, evaluate, and explain models without programming. The tool automates model selection and provides insights into feature importance, demonstrated effectively on the Titanic dataset for survival prediction and feature reduction.
Understanding how complex Machine Learning (ML) models arrive at their decisions is crucial for building trust and fostering collaboration between humans and AI. This challenge is particularly evident in interdisciplinary projects where AI researchers need to communicate intricate model outcomes to domain experts who may lack programming knowledge.
A new Python-based tool called SAInT (Sensitivity Analysis in The Loop) aims to bridge this gap. Developed by Manuela Schuler, SAInT provides a visual and interactive way to explore and understand ML model behavior through integrated local and global sensitivity analysis. The tool is designed to support Human-in-the-Loop (HITL) workflows, allowing users – from AI researchers to domain experts – to configure, train, evaluate, and explain models using a graphical interface, without needing to write any code.
SAInT automates key steps such as model training and selection. It offers global feature attribution using variance-based sensitivity analysis and provides per-instance explanations through popular methods like LIME and SHAP. This means users can get both a broad understanding of which features are generally important across the entire dataset (global sensitivity) and a detailed explanation of why a specific prediction was made for an individual data point (local sensitivity).
How SAInT Works
The tool implements a comprehensive HITL workflow for data understanding. Users begin by selecting features and loading their CSV data. SAInT supports classical ML models like RandomForest and XGBoost, as well as Deep Learning models such as Multilayer-Perceptron (MLP) and Tabular ResNets. After training or loading models, they are automatically evaluated, and the best-performing model is selected for further analysis. Users can choose from various loss functions for both regression and classification tasks.
A key strength of SAInT is its interactive visualization. For each output feature, an interactive subplot is generated, showing ground truth and prediction values. Users can select individual data samples within these plots to trigger local explanations using LIME or SHAP. These explanations reveal the positive and negative impact of features on a specific prediction, along with input feature values.
Global sensitivity analysis is automatically performed on the selected best model, calculating the importance of features across the entire data space. This is visualized through a plot showing first and total order Sobol indices for each input feature. This information is invaluable for identifying the most influential features, which can then guide feature selection for subsequent model training iterations.
Also Read:
- Unraveling AI’s Multimodal Decisions: A Review of Explainability in Attention Models
- Understanding AI Grading Uncertainty with Semantic Entropy
Benefits and Applications
SAInT offers several practical applications:
- Hyperparameter Tuning: Users can iteratively adjust model parameters and compare performance to find the best model for their dataset.
- Bias Detection: The tool helps measure the impact of sensitive features (like gender or race) on model predictions, aiding in the identification and addressing of biases present in the training data.
- Gaining Insights: By using local sensitivity analysis on specific samples, users can determine which features are most important for predictions with high or low output values.
- Feature Selection: Global sensitivity analysis helps identify and remove unimportant input features, leading to more focused and potentially better-performing models.
- Outlier Visualization: Users can identify and visualize outliers, which can inform improvements in data generation.
- Model Reliability: The tool allows users to evaluate model behavior across different data scenarios, building trust in the trained model.
The paper demonstrates SAInT’s capabilities using a survival prediction task on the well-known Titanic dataset. The analysis showed that passenger class, sex, and age were the most influential factors for survival. This information was then used to reduce the number of input features, allowing the model to focus on the most impactful variables.
While currently limited to tabular CSV data, SAInT’s data-centric approach makes AI more accessible for interactive data analysis, empowering domain experts to gain insights into their datasets rather than just focusing on the models themselves. For more details, you can refer to the full research paper.
