TLDR: This research paper proposes a novel mechanism for In-context Learning (ICL) in Language Models (LMs): task-oriented information removal. It demonstrates that LMs in zero-shot scenarios encode non-selective representations, leading to arbitrary outputs. However, by explicitly removing task-irrelevant information via low-rank filters or implicitly through few-shot demonstrations, LMs can be steered towards the intended task. The study identifies specific “Denoising Heads” responsible for this information removal, showing their independence from “Induction Heads” and their critical role in maintaining ICL accuracy, especially in unseen label scenarios. This mechanism, primarily a dimensionality reduction, is crucial for classification tasks but not for bijective tasks.
In-context Learning (ICL) has emerged as a powerful few-shot learning method in modern Language Models (LMs), allowing them to perform tasks with minimal examples. However, the exact mechanisms behind its effectiveness have remained somewhat mysterious. This research paper sheds light on this by introducing a novel perspective: task-oriented information removal.
The study begins by observing that in a zero-shot scenario, where no examples are provided, LMs encode queries into broad, non-selective representations. These representations contain information for all possible tasks, leading to arbitrary and often incorrect outputs because the model doesn’t focus on the intended task. For instance, if asked about “Donald Trump” in a zero-shot setting, an LM might output “President” instead of “Male” if the task was to identify gender.
A key finding is that by selectively removing specific, redundant information from these hidden states using a low-rank filter, LMs can be effectively steered towards the intended task. This process significantly boosts accuracy from near-zero to a remarkable level, even when preserving only a tiny fraction (0.7%) of the dimensions. This demonstrates that filtering out task-irrelevant information is crucial for accurate predictions.
Building on this, the researchers found that few-shot ICL intrinsically simulates this task-oriented information removal. When given a few demonstrations, LMs autonomously guide their hidden states towards a specific “Task-Verbalization Subspace” (TVS). This TVS is a low-rank space within the hidden states that concentrates task-related information, while redundant information orthogonal to it is removed. This mechanism helps LMs produce task-specific outputs, and its behavior is consistent even in scenarios where the correct label is not present in the demonstrations (unseen label scenarios), although accuracy is still affected.
The paper identifies specific attention heads responsible for this removal operation, terming them “Denoising Heads.” These heads are distinct from previously identified “Induction Heads,” which primarily copy labels from demonstrations. Denoising Heads re-encode queries’ information, performing task-oriented information removal. Ablation experiments, where these Denoising Heads are disabled, show a significant drop in ICL accuracy, especially in unseen label scenarios where accuracy plummets to near zero. This confirms the critical role of both the information removal mechanism and the Denoising Heads.
Further analysis reveals that Denoising Heads exhibit a local attention pattern, focusing on query tokens to identify and amplify task-relevant information. While some Denoising Heads are common across different tasks, many are dataset-specific, suggesting a hybrid of in-context learning and in-weight learning properties. This indicates that ICL capabilities might stem from latent tasks learned during pre-training and are further refined by demonstrations.
Also Read:
- SIM-CoT: Enhancing LLM Efficiency and Accuracy Through Supervised Implicit Thoughts
- Finding the Right Dose: A Scaling Law for Knowledge Infusion in LLMs
However, this task-oriented information removal, which is essentially a dimensionality reduction, is most effective for classification tasks that map a broad input space to a narrow label space. It does not apply to bijective tasks like country-capitals, where information is not removed, and the input can be losslessly reconstructed from the output. This distinction highlights the specific types of tasks where this mechanism is most relevant. For more details, you can refer to the original research paper: Mechanism of Task-Oriented Information Removal in In-Context Learning.
