TLDR: The RAISE (Responsible AI Scoring and Evaluation) framework addresses the critical need for quantitative verification of responsible AI principles beyond predictive accuracy. It unifies the evaluation of AI models across four key dimensions: explainability, fairness, robustness, and sustainability. By evaluating deep learning models on real-world datasets, RAISE reveals inherent trade-offs between different architectures—for example, a Transformer’s high explainability and fairness often come at a significant environmental cost, while an MLP might be sustainable and robust but less explainable. The framework emphasizes that responsible AI selection involves choosing a model whose trade-off profile best fits the specific ethical and operational requirements of an application, rather than simply aiming for the highest accuracy.
As artificial intelligence systems become increasingly integrated into critical sectors like finance and healthcare, simply achieving high predictive accuracy is no longer enough. The need for AI models to be explainable, fair, robust, and sustainable has become paramount, driven by both ethical considerations and emerging regulatory frameworks such as the EU AI Act.
However, a significant challenge has been the lack of a unified approach to quantitatively measure these crucial aspects of responsible AI. Existing tools often focus on individual dimensions in isolation, leading to a fragmented understanding of a model’s overall responsibility profile. For instance, a model might be highly explainable but computationally expensive, or fair but vulnerable to adversarial attacks. This fragmentation makes it difficult for practitioners to conduct a holistic, evidence-based risk analysis and select models that truly align with responsible AI principles.
Introducing RAISE: A Unified Framework
To bridge this gap, researchers Loc Phuc Truong Nguyen and Hung Thanh Do have introduced RAISE (Responsible AI Scoring and Evaluation), a novel framework designed to systematically quantify model performance across four foundational dimensions: explainability, fairness, robustness, and sustainability. The framework specifically targets models used with structured (tabular) data, which is prevalent in highly regulated industries.
RAISE’s core innovation lies in its performance-controlled evaluation, which normalizes for predictive F1-Score. This rigorous approach allows for the isolation and comparison of the inherent responsibility profiles of different model architectures, revealing fundamental trade-offs.
The Four Pillars of Responsibility
The framework evaluates models using a comprehensive suite of 21 quantitative metrics, categorized under these four dimensions:
-
Explainability: This dimension assesses how well a model’s predictions can be linked to its input features. RAISE uses SHAP for feature attributions and evaluates their quality using metrics from the Quantus framework, covering aspects like explanation robustness, faithfulness, randomization, and complexity.
-
Fairness: Focusing on mitigating systemic biases, fairness is evaluated by quantifying performance disparities across sensitive subgroups. Metrics include absolute differences in standard classification metrics (Accuracy, Precision, Recall, False Positive Rate) and formal group fairness measures like Demographic Parity and Equalized Odds from toolkits like AIF360 and Fairlearn.
-
Sustainability: This pillar addresses the environmental and resource costs of AI. RAISE estimates carbon emissions (CO2e) using the Lacoste Score and measures computational efficiency through parameters count, FLOPs, and MACs.
-
Robustness: Measuring a model’s ability to maintain performance under non-ideal conditions, such as adversarial attacks. Metrics include the FGSM Accuracy Gap, CLEVER-u Score, and Loss Sensitivity, implemented with the Adversarial Robustness Toolbox (ART).
From Metrics to a Responsibility Score
Each raw metric within RAISE is normalized, and these normalized values are then averaged to produce a Dimension Score for each of the four pillars. While the framework can compute a single, aggregated Responsibility Score for a high-level summary, it emphasizes the multi-dimensional responsibility profile as a more informative tool for nuanced decision-making. Predictive accuracy is reported separately to facilitate a direct analysis of trade-offs.
Experimental Insights and Key Trade-offs
The researchers evaluated three deep learning models—a Multilayer Perceptron (MLP), a Tabular ResNet, and a Feature Tokenizer Transformer—on structured datasets from finance (German Credit), healthcare (Diabetes), and socioeconomics (ACSIncome). All models were trained to achieve comparable F1-Scores, ensuring a fair comparison of their responsibility profiles.
The findings revealed critical trade-offs:
-
The **Multilayer Perceptron (MLP)** demonstrated strong sustainability and robustness, being energy-efficient and relatively stable against perturbations. However, its explanations were broader and less faithful.
-
The **Feature Tokenizer Transformer** excelled in explainability and fairness, particularly in low-data settings, but came with a significantly high environmental cost, indicating poor sustainability.
-
The **Tabular ResNet** offered a consistently balanced profile, serving as a reliable middle ground across all responsibility criteria.
These results underscore a crucial point: no single model architecture dominates across all responsibility criteria. The study highlights that predictive accuracy alone is a weak and often misleading indicator of a model’s true operational and ethical fitness. Instead, responsible AI selection requires a careful choice of an architectural profile whose built-in trade-offs best match the specific ethical and operational needs of the target application.
Also Read:
- How AI Thinks Morally: A Deep Dive into the MOREBENCH Evaluation
- C-SWAP: A New Approach to Efficient Neural Network Compression Using Explainable AI
Future Directions
RAISE provides a practical and modular foundation for evidence-based governance in AI. Future work will expand the framework to include classic non-neural models, add privacy as a core dimension, and conduct usability studies with stakeholders to validate its effectiveness in real-world workflows. The implementation of RAISE is publicly available for further exploration and use at its GitHub repository. You can read the full research paper here.
