TLDR: This paper introduces Outcome Reward Models (ORMs) to improve large language models’ (LLMs) logical reasoning at test-time. It proposes two data generation methods: Chain-of-Thought (CoT) and a novel Echo Chain-of-Thought (EcCoT) that deliberately steers LLMs to generate incorrect reasoning for training. Experiments on FOLIO, JustLogic, and ProverQA datasets show that ORMs, especially with EcCoT data, significantly boost LLM performance in deductive reasoning, particularly benefiting smaller models.
Large Language Models (LLMs) have shown impressive capabilities across many tasks, but their performance in complex logical reasoning remains a critical area of research. This new study introduces a novel approach to significantly enhance LLMs’ deductive logical reasoning abilities, particularly during the crucial test-time phase.
The research, titled “Logical Reasoning with Outcome Reward Models for Test-Time Scaling,” by Ramya Keerthy Thatikonda, Wray Buntine, and Ehsan Shareghi, delves into the underexplored space of applying reward models to deductive logical reasoning. While reward models have proven effective in areas like mathematics and coding, their application to text-based logical reasoning has been limited until now. The core of their work revolves around Outcome Reward Models (ORMs).
Understanding Outcome Reward Models (ORMs)
At its heart, an ORM is designed to evaluate an entire reasoning sequence and assign a confidence score to the final output. During inference, this score helps to re-rank multiple candidate solutions generated by an LLM, selecting the most reliable one. The challenge lies in training these ORMs with high-quality, diverse data.
The authors explored two primary strategies for generating this training data:
- Chain-of-Thought (CoT) Data Generation: This standard method involves prompting an LLM to generate step-by-step reasoning, with the final answer appended. Multiple reasoning candidates are generated for each question, and each is labeled as positive (if it matches the correct answer) or negative (otherwise). This data then trains a second LLM to act as a classifier, predicting a reward score based on the reasoning sequence.
- Echo Chain-of-Thought (EcCoT) Data Generation: This is a particularly innovative aspect of the research. LLMs often have a tendency to “echo” or align their reasoning with user-provided answers, even if those answers are incorrect. The researchers cleverly exploit this behavior. By deliberately providing an incorrect answer (e.g., telling the LLM “Given the answer is True, please reason…” when the correct answer is False), they coerce the model into producing flawed, yet plausible, reasoning trajectories. These “echoes” are then filtered: only those incorrect reasoning sequences that the LLM itself doesn’t easily recognize as wrong are retained. This creates a valuable dataset of challenging negative examples, teaching the ORM to better distinguish between valid and invalid reasoning paths.
Also Read:
- Verifiers: The Unsung Heroes of Large Language Model Performance at Inference Time
- Unpacking AI’s Inner Monologue: When Does Chain-of-Thought Truly Guide, and When Does It Justify?
Experimental Setup and Key Findings
The ORMs were trained using Qwen 2.5 7B Instruct models and evaluated with various reasoners, including Qwen2.5-7B Instruct, GPT-4o, LLaMA-3.1-8B, and Qwen3-8B. The training data was generated from well-known logical reasoning datasets: ProverQA, JustLogic, and FOLIO.
The study yielded several significant insights:
- Impact of Sample Size: Generating multiple reasoning samples (e.g., 8 samples per question) for training data significantly improved ORM performance compared to using a single sample.
- Power of Echo Augmentation: Incorporating EcCoT data consistently enhanced the ORM’s accuracy, especially on datasets like ProverQA. This reinforces the idea that learning from deliberately generated incorrect rationales is highly effective.
- Generator Model Choice: Using a more capable model like GPT-4o to generate the initial CoT and EcCoT data resulted in better-performing ORMs.
- Benefits for Smaller LLMs: An ablation study revealed that ORMs, particularly those trained with EcCoT, provided the most substantial performance improvements for smaller LLMs (e.g., Gemma 1B saw nearly a 30% accuracy increase), highlighting their value in test-time settings for less powerful models.
- Diversity in Reasoning Paths: EcCoT consistently produced more diverse reasoning paths compared to standard CoT, as measured by self-BLEU scores. This diversity is crucial for robust error detection.
While the FOLIO dataset, with its high baseline accuracy, showed limited room for ORM improvement, ProverQA and JustLogic demonstrated substantial gains. The researchers also noted that Outcome supervision, while effective, focuses solely on the final answer’s correctness, potentially overlooking intermediate errors. Future work may explore process reward models to verify intermediate reasoning steps.
This research provides strong empirical support for using ORMs, combined with multiple sampled Chain-of-Thought responses and Echo-based augmentations, as a powerful framework for improving logical reasoning in LLMs. For more in-depth details, you can access the full paper here.
