TLDR: This paper challenges the common belief that the first layers of Large Language Models (LLMs) are most informative for understanding how training data influences their decisions. It demonstrates that the ‘cancellation effect’ is an unreliable metric and that middle attention layers often provide better insights. The authors also introduce novel methods for combining influence scores across layers (Rank and Vote), which significantly improve performance, and propose new metrics (Noise Detection Rate and AUC) for efficiently evaluating influence functions without expensive model retraining.
Understanding how Large Language Models (LLMs) make decisions is crucial for building trust, ensuring safety, and identifying biases in these powerful AI systems. One key approach to this is through ‘influence functions,’ which help pinpoint how individual training data samples impact the model’s outputs.
However, applying influence functions to today’s massive LLMs, with billions of parameters, is computationally very expensive. To make it feasible, researchers often focus on a subset of the model’s layers. Previously, a widely cited work by Yeh et al. (2022) suggested that the first, or ’embedding,’ layers were the most informative for this task, based on a concept called the ‘cancellation effect,’ where opposing influence scores might cancel each other out in later layers.
Challenging the Status Quo: The Unreliable Cancellation Effect
This new research, titled First is Not Really Better Than Last: EVALUATINGLAYERCHOICE ANDAGGREGATIONSTRATEGIES INLANGUAGEMODELDATAINFLUENCEESTIMATION by Dmytro Vitel and Anshuman Chhabra, directly challenges this long-held assumption. Through both theoretical analysis and extensive experiments, the authors demonstrate that the cancellation effect is actually an unreliable indicator of a layer’s usefulness for influence estimation. They even show that including layers with a high cancellation effect can, in some cases, improve the accuracy of influence estimates, contrary to prior beliefs.
Discovering the Most Informative Layers
So, if the first layers aren’t always the best, which ones are? The paper’s findings reveal a consistent pattern: for larger LLMs like Llama-3.2, Qwen-2.5, and Mistral 7B, the ‘middle attention layers’ often provide the most effective influence scores for detecting problematic training samples. The classification head (CL) layer, on the other hand, consistently showed limited ability to identify noise across various models and methods. This suggests that these middle layers capture the most relevant information for understanding how data impacts model behavior.
Smarter Ways to Combine Influence Scores
Beyond identifying the best layers, the researchers also tackled the challenge of combining influence scores from multiple layers. Traditionally, scores are simply averaged, but this can hide important details or allow opposing contributions to cancel each other out. The authors introduce two novel aggregation strategies: ‘Rank’ and ‘Vote.’ These methods, which involve ranking training samples by their influence and assigning votes, significantly outperform standard averaging. The ‘Vote’ method, in particular, showed consistent improvements across multiple datasets and models, highlighting that how influence scores are combined is a critical factor in effective analysis.
Efficiently Evaluating Influence Functions
Evaluating the effectiveness of new influence methods typically requires computationally expensive model retraining. To address this, the paper proposes new, more efficient proxy metrics: the ‘Noise Detection Rate’ (NDR) and ‘Area Under the Curve’ (AUC). NDR measures the proportion of mislabeled samples identified among the least influential ones. The study found that NDR, especially when combined with the ‘Vote’ aggregation strategy, shows a strong correlation with actual downstream performance, offering a reliable way to assess influence functions without the need for costly fine-tuning experiments.
Also Read:
- Unpacking Self-Awareness in Large Language Models
- Streamlining Mechanistic Interpretability with Accelerated Path Patching
A New Path for LLM Interpretability
In conclusion, this research provides significant advancements in understanding and leveraging influence functions for LLMs. It overturns previous assumptions about the most informative layers, demonstrating the superior performance of middle attention layers. It introduces innovative aggregation strategies that enhance the accuracy of influence analysis and offers practical, cost-effective metrics for evaluating these methods. These findings are crucial for developing more trustworthy, transparent, and robust large language models.
