TLDR: This paper proposes a new method for evaluating the quality of explanations (rationales) generated by large language models (LLMs). Instead of just asking humans to pick a preferred explanation, the researchers identify 12 specific attributes (like correctness, plausibility, and completeness) and use them for a more detailed assessment. They found that correctness, plausibility, and completeness are the most important attributes for human preference. This fine-grained evaluation reveals specific strengths and weaknesses of LLMs that binary preference judgments miss, suggesting a path for more interpretable and reliable LLM evaluation and improvement.
Large Language Models (LLMs) have become incredibly powerful, capable of tackling complex tasks that require multi-step reasoning, like solving logical and mathematical problems. A key aspect of their utility is their ability to generate natural language rationales – free-form explanations that not only help improve their performance but also make their reasoning more understandable to human users. Think of these rationales as the step-by-step thought process an LLM provides to arrive at an answer.
However, evaluating the quality of these rationales has been a significant challenge. Traditionally, researchers have relied on binary preference judgments, where humans or even other LLMs simply choose which of two rationales they prefer. While straightforward, this approach is often criticized for being opaque and coarse-grained. It tells us which rationale is ‘better’ but offers little insight into *why* it’s better, or what specific qualities contribute to its superiority.
A new research paper, Rethinking Human Preference Evaluation of LLM Rationales, by Ziang Li, Manasi Ganti, Zixian Ma, Helena Vasconcelos, Qijia He, and Ranjay Krishna, aims to address these limitations. The authors propose a more nuanced approach to evaluating LLM-generated explanations by asking three fundamental questions: What attributes define good rationales? Can human preferences be explained by these attributes? And can attribute-based evaluation overcome the shortcomings of simple binary comparisons?
Identifying Key Rationale Attributes
To answer the first question, the researchers conducted a thorough review of existing literature and identified a set of 12 core attributes that characterize high-quality rationales. These include:
- Faithfulness: Is the rationale supported by the model’s actual computation or evidence?
- Hallucination: Does it introduce information not present in the source?
- Repetition: Does it unnecessarily repeat points?
- Informativeness: Does it add meaningful details?
- Plausibility: Does it ‘sound right’ or seem believable?
- Self-Consistency: Are all reasoning steps logically aligned internally?
- Source Consistency: Does it avoid contradicting the given context?
- Grammar: Is it well-written and free of mistakes?
- Arithmetic Accuracy: Are any calculations correct?
- Conciseness: Is it as short as possible without losing information?
- Completeness: Does it explain all necessary steps?
- Correctness: Are all steps and answers objectively correct?
These attributes were measured using three methods: automated metrics, LLM judges (both open- and closed-source models like GPT-4o and Gemini 2.5-Flash), and human annotations.
Explaining Human Preferences with Attributes
To understand if these attributes could explain human preferences, the team applied SHAP (SHapley Additive exPlanations) analysis to a machine learning model. This technique helps identify which attributes significantly influence human judgment. They used two widely-used human preference datasets, MT-Bench and Chatbot Arena, treating human annotations as the gold standard.
The findings were insightful: Correctness, Plausibility, and Completeness emerged as the most predictive attributes of human preference across different models and datasets. This suggests that when humans judge a rationale, they primarily value factual accuracy, believability, and thoroughness.
More Informative Evaluations with Attribute-Specific ELO Scores
Moving beyond binary comparisons, the researchers introduced attribute-specific ELO scores. ELO ratings are commonly used in competitive games to rank players. Here, instead of a simple win/loss based on overall preference, models were ranked for each individual attribute based on their LLM judge scores. This approach revealed a more nuanced picture of model capabilities.
While top models like GPT-4, GPT-3.5-Turbo, and Claude-v1 generally maintained their leading positions, the attribute-specific ELO scores uncovered specific strengths and weaknesses. For instance, Claude-v1 struggled with Repetition, and surprisingly, GPT-3.5-Turbo outperformed GPT-4 in Arithmetic Accuracy and Self-Consistency on certain benchmarks. These subtle differences are often obscured by a single, holistic preference score.
Also Read:
- Assessing Foundation Models for Planning Assistance
- Quantifying and Refining Large Language Model Performance in Academic Proposal Writing
Practical Recommendations and Future Directions
Based on their findings, the authors recommend that future research in rationale evaluation should move beyond binary preferences towards fine-grained, attribute-level assessments. They suggest focusing on attributes most predictive of human preference, such as Correctness, Plausibility, Completeness, Informativeness, and Conciseness. While LLM judges can be a scalable solution for this, they advise caution, recommending the use of multiple LLM judges and transparent reporting of their outputs to mitigate potential biases.
This work represents a significant step towards developing more interpretable and reliable evaluation practices for LLM-generated rationales. By dissecting human preference into its constituent attributes, researchers can gain a deeper understanding of what makes an explanation truly effective and guide the development of more transparent and trustworthy LLMs.
