TLDR: This research introduces new sampling techniques, feature-based and logit-based, to improve the accuracy and efficiency of influence functions for explaining black-box AI models. Traditional random sampling is inconsistent. The proposed methods select more representative data subsets, significantly reducing computation time and memory usage (up to 30.1% and 42.2% respectively) and boosting F1-score accuracy by 2.5% in class removal tasks, making AI explainability more practical.
Understanding how artificial intelligence models make their decisions, especially those considered “black-box” due to their complexity, is becoming increasingly vital. This need for transparency is crucial for ensuring AI systems align with human ethics and societal values. One powerful tool for this purpose is the influence function, which helps explain how individual training data points affect a model’s parameters.
Traditionally, influence functions offer a way to evaluate the impact of data without the costly process of retraining a model from scratch. They do this by leveraging gradients and Hessians, mathematical components that describe how a model’s output changes with respect to its inputs and parameters. However, a significant hurdle has been the immense computational and memory resources required to calculate the Hessian for an entire dataset, particularly with the rise of large-scale AI models like large language models (LLMs).
A common workaround involves randomly sampling a small subset of the training data to estimate the Hessian. While this reduces the computational burden, it often leads to highly inconsistent and inaccurate influence estimates. The problem lies in the high variance introduced by these random samples, meaning different random selections can yield vastly different results, making the explanations unreliable.
Introducing Advanced Sampling Techniques
To overcome these limitations, researchers Jungyeon Koh, Hyeonsu Lyu, Jonggyu Jang, and Hyun Jong Yang propose two innovative advanced sampling techniques: feature-based sampling and logit-based sampling. These methods aim to select a small yet highly representative subset of the entire dataset, thereby significantly improving the accuracy and consistency of influence function estimations.
Feature-based sampling works by organizing data points within a latent feature space. The paper explores both extrinsic (using a pre-trained Vision Transformer) and intrinsic (using the model under investigation) feature extractors. Within this feature space, two sampling methods are applied:
- Top-k sampling: Identifies centroids using the K-means algorithm and selects the ‘k’ nearest samples to each centroid.
- Distance-weighted sampling: Creates a probability distribution based on the distance of features from centroids, allowing for a degree of stochasticity while still prioritizing representative samples.
Logit-based sampling takes a different approach, focusing on the class-wise softmax scores (logits) of each data point. For each class, it creates a multinomial distribution based on these scores and then selects ‘k’ samples from this distribution. This method effectively captures data points that are highly representative of their respective classes.
Demonstrated Improvements
The efficacy of these new sampling methods was validated through class removal experiments, a typical application of influence functions. Using metrics like F1-score (to measure how effectively the model forgets a removed class while maintaining performance on others), run-time efficiency, and memory efficiency, the results were compelling.
The logit-based sampler consistently yielded the most accurate estimations, outperforming other methods. It also proved to be the most computationally efficient, requiring the least compute cost as it directly uses the softmax results without needing external neural networks or K-means algorithms. Distance-weighted samplers also showed strong performance, comparable to the logit method.
Notably, the proposed methods demonstrated significant improvements in efficiency. Compared to the baseline random sampling, the new techniques reduced computation time by up to 30.1% and memory usage by 42.2%. Furthermore, they improved the F1-score by 2.5%, indicating more effective model unlearning and better overall performance. For instance, to achieve the same F1-score as the best random sampler (requiring 1,300 samples), the logit and intrinsic top-k samplers only needed 900 and 1,100 samples respectively, leading to substantial savings in time and memory.
Also Read:
- Unveiling Agent Influence in Collaborative AI Workflows with CAIR
- Judo Calculus: A New Framework for Context-Aware Causal Discovery
Towards More Transparent AI
This research marks a significant step forward in making influence functions more practical and reliable for understanding complex AI models. By addressing the computational and accuracy limitations of traditional sampling, these advanced techniques pave the way for more robust AI systems through a deeper, bottom-up understanding of their internal mechanisms. The full research paper can be accessed here.
