TLDR: A new research paper introduces a human-LLM collaborative framework to infer ‘thinking traces’—the reasoning behind human judgments—from label-only annotations. By using rejection sampling with Reasoning Language Models, these traces are reconstructed at scale. The inferred traces are then used in two ways: to fine-tune open LLM raters for better alignment with human judgments, and to synthesize clearer annotation guidelines for proprietary LLM raters. Both methods significantly improve LLM-human agreement and inter-LLM reliability, demonstrating that LLMs can act as proxies for human thinking to enhance the dependability of automated evaluators.
Large language models, or LLMs, are becoming increasingly common as automated evaluators for various tasks, especially those involving open-ended text generation. They offer remarkable efficiency and scalability, making them attractive for applications like evaluating code, text summaries, and dialogue. However, a significant challenge arises when these models are used for subjective tasks, where human judgments often involve subtle reasoning that goes beyond simple labels.
Imagine training a human annotator for a complex qualitative task. They would benefit not just from a detailed instruction manual (codebook) and examples, but crucially, from explanations of the reasoning behind specific judgments. These explanations, which researchers call ‘thinking traces,’ are vital for creating a shared understanding and ensuring consistent evaluations. Unfortunately, collecting these detailed thinking traces from humans is time-consuming and costly, leading many datasets to lack this rich context.
A new research paper, Through the Judge’s Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters, introduces a novel human-LLM collaborative framework designed to infer these valuable thinking traces from existing label-only annotations. The core idea is to leverage Reasoning Language Models (RLMs), which are designed to generate step-by-step reasoning, as a proxy for human thought processes. The framework uses a simple yet effective rejection sampling method: for each input, the RLM generates multiple candidate thinking traces and their corresponding labels. Only those traces where the RLM’s predicted label matches the human’s original label are kept. This process reconstructs thinking traces at scale, transforming basic datasets into rich, reasoning-augmented resources.
The utility of these inferred thinking traces is demonstrated through two complementary applications. The first involves fine-tuning open-weight LLM raters. By training these models not just on input-label pairs but on the complete reasoning trace (input, thinking trace, label), the models learn not only what label to produce but also how to reason towards it. This ‘reasoning-enhanced supervised fine-tuning’ significantly improves the alignment of LLM raters with human judgments across various tasks, including evaluating short story complexity, student essays, news summaries, translations, and chatbot responses.
The second application addresses proprietary LLMs, where direct fine-tuning is not an option. For these black-box models, the inferred thinking traces are used to automatically refine annotation codebooks. Many existing codebooks suffer from ambiguous descriptions or a lack of concrete criteria, leading to inconsistent ratings. Inspired by iterative refinement practices in qualitative coding, the framework extracts common reasoning patterns from the inferred traces to synthesize clearer, more explicit, step-by-step guidelines. This refined codebook then serves as a more effective prompt to guide the proprietary LLM’s behavior. Experiments show that these refined codebooks not only improve agreement between LLM raters and human judgments but also boost inter-rater reliability among different LLM models, promoting cross-model consistency.
The research highlights that the refinement process works in two stages: improving task instructions and enriching scoring rubrics. Instruction refinement provides a clear, standardized step-by-step procedure, while rubric refinement grounds descriptions in concrete examples, helping models converge on nuanced conclusions. An ablation study confirmed that both components contribute to performance enhancement, with their relative impact depending on the original codebook’s specific weaknesses.
Also Read:
- Asynchronous Thinking: Language Models Learn to Organize for Collaborative Problem Solving
- Bridging the Gap: A New AI System Learns to Aggregate Diverse Human Preferences
This work suggests that LLMs can effectively serve as practical proxies for otherwise unrevealed human thinking traces. By extending label-only corpora into thinking-trace-augmented resources, the reliability and interpretability of LLM-based evaluators can be significantly enhanced. This opens doors for more robust automated evaluation systems and even potential future human-AI collaboration in annotation tasks, where humans could validate AI-generated traces rather than creating them from scratch.
