Unpacking Moral Bias in Language Models: A Layer-by-Layer Look

TLDR: Research by Raimondi, Dalbagno, and Gabbrielli investigates the Knobe effect, a moral bias, in finetuned large language models (LLMs). They found that LLMs, particularly after finetuning, exhibit this bias where negative outcomes are judged as more intentional. Crucially, the bias is localized to specific mid-to-late Transformer layers. The study successfully demonstrated that this bias can be significantly mitigated using a Layer-Patching technique, which replaces biased activations with those from the unbiased pretrained model, offering a path for targeted bias reduction without retraining.

Large language models (LLMs) are becoming increasingly integrated into various aspects of our lives, from decision-making to ethical reasoning. As their influence grows, it becomes crucial to understand if these powerful AI systems internalize human-like biases. A recent research paper delves into this very question, specifically investigating a moral bias known as the Knobe effect in finetuned LLMs and exploring how it can be understood and even mitigated.

Understanding Moral Bias in LLMs

The study, titled “Analysing Moral Bias in Finetuned LLMs through Mechanistic Interpretability,” was conducted by Bianca Raimondi, Daniela Dalbagno, and Maurizio Gabbrielli from the University of Bologna. Their work focuses on the Knobe effect, a well-documented cognitive bias in human moral judgment. This effect demonstrates that people are more likely to judge an action as intentional when it leads to a negative side effect than when it results in a positive one, even if the agent’s actual intention remains the same.

The Knobe Effect Explained

Imagine a scenario where a company CEO implements a new policy. If the policy leads to a positive outcome, people might say the CEO didn’t intentionally cause the good, but if it leads to a negative outcome, they might say the CEO intentionally caused the harm, even if the CEO’s initial intent was neutral or focused solely on profit. This asymmetry in judging intentionality based on the outcome’s moral valence is the Knobe effect.

Investigating the Bias: Research Questions

The researchers set out to answer three key questions:

Do LLMs exhibit the Knobe effect, and if so, is it a consequence of finetuning?
Are specific layers within the Transformer architecture of LLMs responsible for encoding this bias?
Can targeted interventions, such as a technique called Layer-Patching, be used to reduce or eliminate this bias?

How the Study Was Conducted

To explore these questions, the team evaluated three popular open-weights LLM families: Llama, Mistral, and Gemma, including various model sizes. They used a dataset of moral scenarios, similar to those used in human psychological studies, where models were prompted to rate the intentionality of an action on a scale of 0 to 10, given either a positive or negative side effect.

Detecting the Bias

For the first research question, they compared the responses of both pretrained (base) and finetuned (instruction-aligned) versions of the models. They generated multiple responses for each scenario, introducing a degree of randomness to simulate human variability. The Knobe effect was quantified as the difference in average intentionality scores between negative and positive outcome scenarios.

Pinpointing the Bias Location

To understand where the bias might be encoded within the model, the researchers used a technique to analyze the internal “residual stream activations” of each Transformer layer. By comparing these activations between negative and positive scenarios in both pretrained and finetuned models, they aimed to identify specific layers where the moral bias became apparent.

Reducing the Bias with Layer-Patching

For the third question, they developed a Layer-Patching algorithm. This innovative method involves selectively replacing the internal activations of a specific layer in the finetuned (biased) model with the corresponding activations from its pretrained (unbiased) version. The goal was to see if this targeted intervention could reduce the Knobe effect without needing to retrain the entire model.

Key Findings

Finetuning Amplifies Bias

The study found that pretrained LLMs showed little to no Knobe effect, meaning their intentionality judgments were largely consistent regardless of the outcome’s moral valence. However, after finetuning, all tested models consistently exhibited a strong Knobe effect, similar to human behavior. This indicates that finetuning, while designed to align models with human preferences, also causes them to internalize human biases.

Bias Resides in Specific Layers

Crucially, the researchers discovered that the Knobe effect was not diffusely spread throughout the models but was localized to specific mid-to-late Transformer layers in the finetuned versions. This suggests that finetuning introduces structural representations of moral bias within these particular parts of the model’s architecture.

Targeted Intervention Works

The Layer-Patching technique proved highly effective. By replacing the activations of just a few critical layers in the finetuned models with those from their pretrained counterparts, the Knobe effect was significantly reduced, almost to zero in some cases. This demonstrates that the moral bias is not only localizable but also reversible through targeted interventions without altering model parameters or requiring further training.

Model Size and Bias

An ablation study also revealed that the Knobe effect is primarily induced by finetuning, rather than being an inherent property of model scale alone. While the magnitude of the bias increased with larger models, it consistently emerged after finetuning across all sizes, reinforcing the idea that alignment data and objectives are key drivers of this bias.

Also Read:

Implications and Future Directions

These findings challenge the notion that social biases in LLMs are merely emergent and diffusely represented. Instead, the research provides concrete evidence that moral biases can be mechanistically localized to specific computational modules within the model. This opens up exciting possibilities for diagnosing and repairing biases in LLMs with minimal disruption to their overall performance.

The study highlights mechanistic interpretability not just as a diagnostic tool but as a practical method for model repair. Future work could explore if this localization technique applies to other types of biases, such as gender or racial stereotypes, and develop methods that don’t require access to the original pretrained model. This research marks a significant step towards building more responsible and ethically aligned AI systems.

For more in-depth information, you can read the full research paper here.