TLDR: A new method called WIMHF uses sparse autoencoders to automatically identify and describe human-interpretable features within language model preference data. This helps explain what preferences datasets measure and what annotators actually prefer, revealing diverse, sometimes unsafe, preferences across datasets. The insights enable better data curation for safety and fine-grained personalization of language models.
Human feedback is crucial for training language models, but understanding exactly what this feedback encodes can be challenging. This lack of clarity often leads to unpredictable and sometimes undesirable model behaviors. While previous research has looked at specific attributes like length or sycophancy, automatically extracting relevant features without pre-defining hypotheses has remained a significant hurdle.
A new method, called What’s In My Human Feedback? (WIMHF), aims to address this by explaining feedback data using sparse autoencoders. WIMHF helps characterize two key aspects: first, the preferences a dataset is capable of measuring, and second, the preferences that annotators actually express.
The WIMHF method operates in three main steps. Initially, it trains a sparse autoencoder (SAE) to learn interpretable features from the differences between pairs of responses. These are referred to as “measurable preferences.” Next, it generates natural language descriptions for each of these features using a large language model. Finally, it identifies “expressed preferences” by determining which features predict the human preference labels, while also accounting for other factors like response length.
The researchers applied WIMHF across seven different datasets, including LMArena, Community Alignment, HH-RLHF, PRISM, and Reddit. Their findings indicate that WIMHF can identify a small number of human-interpretable features that account for a significant portion of the preference prediction signal achieved by more complex, black-box models.
One of the key insights from WIMHF is the wide diversity in human preferences and the influence of dataset-level context. For example, users on Reddit tend to prefer informality and jokes, whereas annotators in HH-RLHF and PRISM datasets disfavor them. WIMHF also brought to light potentially unsafe preferences, such as the tendency of LMArena users to vote against refusals, often in favor of toxic content.
The features learned through WIMHF have practical applications. They enable effective data curation; for instance, re-labeling harmful examples in the Arena dataset led to substantial safety improvements (a 37% gain) without compromising general performance. Furthermore, these features allow for fine-grained personalization. On the Community Alignment dataset, WIMHF learned annotator-specific weights for subjective features, which improved preference prediction. The most subjective preference identified was the choice between paragraphs and bulleted lists in responses.
The method’s validity was confirmed through several evaluations, including its accuracy in predicting preference labels, its alignment with annotator-written explanations, and qualitative validation by external machine learning researchers. WIMHF provides a human-centered analysis tool for practitioners to gain a clearer understanding of preference data, which can help in anticipating and controlling how language models behave during the finetuning process. It also highlights the potential conflicts that can arise when mixing datasets with differing human preferences.
Also Read:
- Sparsity and Specialization: Making Sense of Mixture of Experts Models
- Efficient LLM Evaluation: A New Item-Centric Approach with Cognitive Scales
For more details, you can read the full research paper: What’s In My Human Feedback? Learning Interpretable Descriptions of Preference Data.
