TLDR: A new research paper demonstrates that finetuning Large Language Models (LLMs) on narrow domains leaves strong, readable biases in their internal activations. Using “model diffing” techniques and an AI interpretability agent, researchers successfully identified finetuning objectives with high accuracy. These biases, likely a form of overfitting, can be mitigated by mixing pretraining data, but this raises questions about the realism of narrowly finetuned models as research proxies.
Large Language Models (LLMs) have become incredibly versatile, adapting to specific tasks through a process called finetuning. This involves further training a general-purpose LLM on a smaller, specialized dataset. While powerful, understanding exactly how this finetuning changes the model’s internal workings has been a complex challenge.
A new research paper, “NARROWFINETUNINGLEAVESCLEARLYREADABLE TRACES INACTIVATIONDIFFERENCES” by Julian Minder and colleagues, sheds light on this mystery. The paper reveals that when LLMs are finetuned on very specific, or “narrow,” domains, they develop strong, interpretable biases in their internal “activations.” These activations are essentially the internal signals or thought processes of the LLM as it processes information.
Uncovering Hidden Biases
The researchers used a set of techniques inspired by “model diffing,” which is the study of differences between a model before and after finetuning. Their primary method, called Activation Difference Lens (ADL), involved comparing the activations of the original (base) model with the finetuned model. They did this by feeding both models random text and observing the differences in their internal responses, particularly on the first few tokens of the text.
Two key tools were employed: Patchscope and Logit Lens. These tools help translate the complex internal activation differences into understandable terms, often revealing specific tokens or concepts related to the finetuning domain. For instance, a model finetuned on baking recipes might show activation differences that highlight words like “bake” or “cake.”
Beyond just observing, the team also “steered” the finetuned models. By subtly adding these activation differences back into the model’s processing, they could make the model generate text that closely resembled the format and content of its original finetuning data, even when given unrelated prompts. This demonstrated that these biases are not just detectable but can actively influence the model’s output.
An AI Agent for Interpretation
To objectively validate their findings, the researchers developed an LLM-based interpretability agent. This agent was tasked with identifying the finetuning objective of various models. When this agent was given access to the “bias insights” derived from activation differences and steering, it performed significantly better than baseline agents that only used standard prompting. It was more than twice as good at identifying the broad finetuning objective and over 30 times better at pinpointing specific details.
The study covered a wide range of finetuning scenarios and model architectures, including Gemma, LLaMA, and Qwen models ranging from 1 billion to 32 billion parameters. They investigated models finetuned for tasks like generating false facts, exhibiting emergent misalignment, subliminal learning (where models acquire preferences through hidden signals), and even taboo word guessing games.
Why These Biases Occur and How to Mitigate Them
The researchers suspect that these strong biases are a form of “overfitting,” where the model becomes too specialized to its narrow training data. They found that if the finetuning data is too “mono-semantic” (meaning all samples share a very similar concept), the model tends to learn this constant bias directly.
Interestingly, they discovered a potential mitigation strategy: mixing pretraining data (general, diverse data) into the finetuning corpus. This approach largely helped to remove the detectable biases. However, this often came with a trade-off, sometimes impairing the model’s ability to fully internalize the specific target objective of the finetuning.
Also Read:
- Uncovering Hidden Biases in Large Language Models with Adaptive Question Generation
- Unmasking AI’s Dark Side: How LLMs Can Be Coerced into Multi-Turn Harassment
Implications for AI Safety and Research
This work has several important implications. Firstly, it provides new ways to understand how finetuned models operate and suggests improvements for their training. Secondly, it serves as a warning to AI safety and interpretability researchers. Many studies use narrowly finetuned models as “model organisms” to simulate behaviors that might arise in more broadly finetuned models (like chat-tuned models). The paper argues that these model organisms might not be realistic proxies because the strong, detectable biases from narrow finetuning can overpower signals from broader training.
The findings highlight a critical need for deeper investigation into the effects of narrow finetuning and the development of more realistic case studies for research in model diffing, safety, and interpretability. For more technical details, you can read the full paper here.
