TLDR: Deep learning models often fail in real-world scenarios due to ‘distributional shifts’ in data, leading to inaccurate performance estimates. This research introduces a novel methodology called Probabilistic Runtime Verification, Evaluation, and Risk Assessment (DL-PRA). It uses ‘event trees’ and ‘out-of-distribution (OOD) detectors’ to continuously estimate the probability of data shifts and the network’s accuracy in real-time. This framework provides more credible performance estimates and enables actionable risk assessments, crucial for safety-critical AI applications, by modeling uncertainty and allowing for cost-benefit analyses.
Deep learning systems, despite their impressive performance on controlled benchmarks, frequently encounter significant challenges when deployed in the real world. A major culprit is what researchers call ‘distributional shifts’ – subtle, often imperceptible changes in input data that cause deep neural networks (DNNs) to underperform. These shifts are common in practical scenarios but are rarely accounted for during initial evaluations, leading to an inflated sense of a system’s true capabilities.
Addressing Real-World AI Performance Gaps
A new methodology, proposed by Birk Torpmann-Hagen, Pål Halvorsen, Michael A. Riegler, and Dag Johansen, aims to bridge this gap. Their work, titled “Probabilistic Runtime Verification, Evaluation and Risk Assessment of Visual Deep Learning Systems,” introduces a robust framework for verifying, evaluating, and assessing the risks of deep learning systems, especially visual ones, as they operate in real-time.
The Core Problem: Distributional Shifts
The paper highlights that DNNs are often trained in an ‘underspecified’ manner, meaning they can learn spurious correlations from training data that don’t hold up in diverse real-world conditions. This leads to poor generalization, where a model performs well on data similar to its training set but fails when faced with new, slightly different data. Examples range from medical imaging (where models might focus on hospital watermarks instead of clinical features) to general image recognition datasets like ImageNet and CIFAR.
A Novel Probabilistic Approach
To tackle this, the researchers propose a methodology inspired by Probabilistic Risk Assessment (PRA), a technique commonly used in high-stakes engineering fields like aerospace and nuclear safety. Central to their approach is the construction of an ‘event tree.’ This tree models potential outcomes for a DNN and their associated probabilities, explicitly accounting for the incidence of distributional shifts at runtime.
The key innovation lies in estimating the probability of these shifts in real-time. This is achieved by using ‘out-of-distribution (OOD) detectors,’ which are auxiliary systems designed to identify when incoming data differs significantly from what the model was trained on. These estimates are then combined with the conditional probabilities of the network making correct predictions under various conditions, all structured within the event tree.
How it Works: Event Trees and Likelihood Estimators
The event tree allows for the computation of credible and precise estimates of network accuracy. It can be configured in different ways: a basic tree considers the probability of correct predictions given a shift, while a more advanced version also accounts for whether the OOD detector successfully identifies the shift, potentially activating fail-safes or flagging data for human review. Each outcome in the tree can also be associated with a cost, enabling comprehensive risk assessment and cost-benefit analyses.
The framework relies on estimating three types of probabilities: event probabilities (e.g., encountering OOD data), conditional detection probabilities (how accurately the OOD detector identifies these events), and conditional accuracy (the DNN’s accuracy given specific events and detection outcomes). The paper introduces a ‘Rogan–Gladen estimator’ to correct for biases in observed shift frequencies, making the estimates more responsive to dynamic environments.
Key Benefits and Real-World Impact
The research demonstrates that this approach consistently outperforms conventional evaluation methods, with accuracy estimation errors typically ranging between 0.01 and 0.1 across five different datasets. This improved accuracy is crucial for safety-critical applications where reliable performance estimates are paramount.
Beyond just accuracy, the framework offers:
- Improved Uncertainty Quantification: Provides continuous, real-time assessment of prediction credibility.
- Resilience to Evolving Data Environments: Adapts to changing data distributions by continuously updating event probability estimates.
- Actionable Risk Assessment: By associating costs with outcomes, it allows for interpretable risk metrics that can inform value-judgments and be communicated to stakeholders.
- Extensibility and Modularity: Can be applied to various adverse data events (like adversarial attacks or prompt violations in large language models) and different data modalities, as long as a corresponding detector can be implemented.
A medical segmentation benchmark, specifically for polyp segmentation, showcased the framework’s potential for risk assessment. By assigning economic costs to different outcomes (e.g., correct detection, necessary intervention, failed detection), the system can inform cost-benefit analyses and help determine maximum tolerable risk levels. This can guide development decisions, showing whether improving the core classifier or the event detector would yield greater risk reduction.
Also Read:
- Enhancing Trust in Medical AI: The Promise of Explainable Uncertainty Estimation
- EvA-RL: Training Reinforcement Learning Policies for Easier and More Accurate Evaluation
Limitations and Future Directions
The authors acknowledge several limitations, including the assumption of a static shift environment within short trace windows, the reliance on sufficiently performant OOD detectors, and the need for OOD calibration datasets that accurately represent real-world OOD data. Future work will explore adaptive trace lengths, more sophisticated OOD detectors, and distinguishing between degrees of accuracy for OOD data. The computational overhead of runtime monitoring is also a consideration, as is extending the framework to other domains like Large Language Models and accounting for multiple, potentially interdependent event types.
Ultimately, this research offers a significant step towards making deep learning systems more trustworthy and reliable, particularly in critical applications, by providing more accurate performance estimates and actionable risk assessments. For more details, you can read the full paper here.
