TLDR: Researchers from Vanderbilt University have developed a new probabilistic verification framework to assess the robustness of Semantic Segmentation Networks (SSNs) against adversarial attacks. This scalable, architecture-agnostic method combines sampling-based reachability analysis with conformal inference, using novel strategies like deflative PCA and surrogate models to reduce conservatism in high-dimensional outputs. It provides reliable safety guarantees and tighter bounds than previous methods, especially for complex models in safety-critical applications like medical imaging and autonomous driving, where traditional deterministic verification often fails.
Deep Neural Networks (DNNs) have transformed many fields, from image classification to medical diagnosis and autonomous systems. However, despite their impressive capabilities, these networks are highly susceptible to small changes in their input, known as adversarial examples. This vulnerability poses significant risks, particularly in safety-critical applications like self-driving cars and medical imaging, where even a minor error can have catastrophic consequences.
The Challenge of AI Robustness
Ensuring the reliability of DNNs has led to two main research directions: falsification-based and verification-based approaches. Falsification methods aim to find adversarial examples that expose a model’s weaknesses, but they don’t offer formal guarantees of safety. Verification methods, on the other hand, try to certify robustness by proving that a network will behave reliably under specific input changes. While these offer strong guarantees, they often struggle with scalability, especially for large and complex models.
Semantic segmentation networks (SSNs), which assign a class label to every pixel in an image, are particularly challenging. Their extremely high-dimensional output spaces and intricate architectures make traditional verification methods computationally infeasible or result in overly conservative safety guarantees.
A Novel Probabilistic Verification Framework
Researchers Navid Hashemi, Samuel Sasaki, Diego Manzanas Lopez, Ipek Oguz, Meiyi Ma, and Taylor T. Johnson from Vanderbilt University have introduced a new probabilistic verification framework designed to overcome these limitations. Their work, detailed in their paper Probabilistic Robustness Analysis in High Dimensional Space: Application to Semantic Segmentation Network, offers a scalable and architecture-agnostic solution for analyzing the robustness of SSNs.
The core of their approach combines sampling-based reachability analysis with conformal inference (CI). Reachability analysis helps understand the range of possible outputs a system can produce given a bounded input. Conformal inference provides statistically sound prediction intervals, offering provable guarantees without making restrictive assumptions about the underlying data distribution.
Addressing High-Dimensionality
A key innovation in this framework is how it tackles the conservatism often associated with conformal inference in high-dimensional settings. The team proposes novel strategies, including the use of surrogate models and deflative principal component analysis (PCA). Instead of directly analyzing the complex output of the SSN, they create a simpler ‘surrogate’ model that approximates the original network. The difference between the original and surrogate model’s predictions (the ‘prediction error’) is then analyzed using conformal inference. Since these errors are typically much smaller in magnitude, this significantly reduces the conservatism of the guarantees.
To further manage high-dimensional outputs, they employ a learning-based deflation algorithm for PCA. This technique iteratively identifies and removes the most dominant principal components, effectively reducing the dimensionality of the data while preserving crucial information. This allows for the construction of more accurate and less conservative ‘reachsets’ – the sets of all possible outputs – which are no longer confined to simple hyper-rectangular shapes but can be represented as more general convex hulls.
Also Read:
- A Formal Framework for Robust Control of AI-Driven Autonomous Vehicles
- Enhancing Autonomous Vehicle Reliability: A New Framework for AI Perception Error Analysis and Control
Practical Applications and Results
The framework also provides a clear method for classifying individual pixels in a segmented image as robust, non-robust, or unknown under adversarial conditions. A pixel is deemed robust if its predicted label remains unchanged across all possible adversarial examples within a specified range. If the label always changes, it’s non-robust. If it changes for some perturbations but not others, it’s unknown.
The researchers empirically evaluated their method on large-scale segmentation models across diverse datasets, including CamVid, OCTA-500, Lung Segmentation, and Cityscapes. Their findings demonstrate that the approach provides reliable safety guarantees while substantially tightening the bounds compared to existing state-of-the-art methods. Furthermore, the technique proved scalable, handling complex models and high perturbation levels where traditional deterministic verification methods often failed due to memory constraints.
This new probabilistic verification method is both scalable and data-efficient. While it doesn’t offer deterministic guarantees, it provides meaningful assurances in scenarios where existing deterministic tools are computationally impractical. The team has also made a toolbox implementing this technique available on GitHub, fostering further research and application in the field of AI safety.
