TLDR: DxHF is a novel user interface that improves the quality of human feedback for aligning large language models (LLMs). Instead of comparing long texts, it breaks them down into individual claims, visually encodes relevance, and links similar claims. Studies show DxHF increases feedback accuracy, especially for uncertain users, though it may slightly increase feedback time.
Aligning large language models (LLMs) with human values is a crucial step in their development, often relying on human feedback. This feedback typically involves human annotators comparing two different text responses generated by an LLM and choosing the better one. However, this process can be cognitively demanding, especially when dealing with long or unfamiliar texts, leading to potential errors and lower quality feedback.
A new research paper, titled “DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decomposition,” introduces a novel approach to tackle this challenge. Authored by Danqing Shi from Aalto University, Furui Cheng and Mennatallah El-Assady from ETH Zürich, and Tino Weinkauf from KTH Royal Institute of Technology, and Antti Oulasvirta from Aalto University, the paper explores the decomposition principle to enhance the quality of human feedback for LLM alignment.
The Decomposition Principle in Action
The core idea behind this research is to break down complex text comparison tasks into simpler, more manageable parts. Instead of asking annotators to compare two lengthy text responses directly, the DxHF (Decomposition for Human Feedback) user interface decomposes these texts into individual, concise statements called “atomic claims.” Each claim represents a single piece of information, making it easier to interpret and compare.
DxHF enhances the comparison process in several ways. It visually encodes the relevance of each claim to the original conversation, using text opacity to de-emphasize less relevant information. This helps users quickly identify and focus on the most important points. Furthermore, DxHF links similar claims across the two responses, often with a keyword summarizing their shared meaning. This feature allows users to skim through key information and pinpoint differences more quickly and accurately.
How DxHF Works
The system first takes the LLM-generated responses and breaks them down into atomic claims using an LLM (GPT-4). These claims are designed to be concise and retain the original meaning without adding new words. Then, each claim is ranked based on its contextual relevance to the user’s query, using a Cross-Encoder architecture. Finally, similar claims from both responses are linked together based on their semantic similarity, and a keyword is generated to summarize the linked claims.
The user interface presents these decomposed claims in two vertical lists, with a central region showing the connecting links. Users can group links by keywords or sort claims by relevance. Interactive hovering allows users to highlight connected claims and their original text parts, providing a focused view while maintaining context. DxHF is also designed to integrate seamlessly into existing workflows, allowing annotators to switch between a full-text view and the decomposed view based on the complexity of the task.
Evaluation and Findings
The researchers conducted three evaluations to assess DxHF’s effectiveness. A technical evaluation using simulated annotators showed that decomposition, especially when combined with ranking and linking, generally improves feedback accuracy across different levels of user rationality. This suggests that the method is particularly beneficial when annotators are uncertain.
A crowdsourcing study with 160 participants further validated these findings. The study revealed that using DxHF improved feedback accuracy by an average of 5% compared to a baseline interface. This improvement was even more significant (6.4% higher accuracy) for participants who expressed less certainty in their answers. While DxHF did increase the average feedback time by about 18 seconds, participants generally found the tool helpful for identifying key information, improving confidence in their decisions, and reducing cognitive load during complex comparisons.
An ablation study with 36 participants evaluated the individual contributions of DxHF’s ranking and linking features. The results indicated that both features are valuable, with ranking helping users focus on key information and linking reducing the effort required for comparison. Participants widely appreciated the combined visual cues and navigational aids provided by DxHF.
Also Read:
- Beyond Relevance: How AI Models Are Learning to Pick Truly Useful Information for Better Answers
- Enhancing Code Review with Fine-Tuned Language Models: A C# Case Study
Implications and Future Directions
The study highlights the significant potential of Human-Computer Interaction (HCI) principles in improving human-AI alignment. By enhancing the quality of human feedback, tools like DxHF can lead to more accurate and less biased LLM training. The researchers acknowledge that while DxHF is effective for factual or task-oriented comparisons, its current design may not be ideal for holistic judgments involving text coherence, tone, or style. They also discuss the importance of ensuring that the decomposition process itself doesn’t introduce new biases.
The interactive decomposition technique presented by DxHF could have broader applications beyond LLM alignment, such as comparing human-written texts or assisting experts in information-intensive reading tasks. The DxHF interface is open-sourced at https://sdq.github.io/DxHF, offering a promising avenue for future HCI and AI research collaboration.
