TLDR: COMPASS is a new self-scoring reward mechanism for Large Language Models (LLMs) that allows them to learn autonomously from unlabeled data. It addresses the limitations of existing methods by combining two components: Dual-Calibration Answer Reward (DCAR), which creates reliable pseudo-labels by assessing confidence and credibility, and Decisive Path Reward (DPR), which optimizes the reasoning process itself by rewarding decisive steps during uncertain moments. This approach significantly improves LLM performance across various reasoning tasks and models, enabling more scalable and robust self-improvement.
Large Language Models (LLMs) have shown incredible potential in complex tasks like mathematics and code generation, largely thanks to Reinforcement Learning (RL). However, a major hurdle for current RL methods is their heavy reliance on vast amounts of human-curated data or labeled datasets to create reward systems. This dependency makes it difficult to scale LLMs, especially as tasks become more intricate and numerous.
To tackle this, researchers are exploring a new approach: enabling LLMs to learn autonomously from continuous streams of unlabeled data. The main challenge here is how to reliably estimate rewards without any ground-truth supervision. Existing methods, such as Test-Time Reinforcement Learning (TTRL), try to solve this by using self-consistent consensus, often through majority voting. The idea is that if an answer is consistently reproduced across multiple attempts, it’s likely correct. However, this method risks reinforcing incorrect answers if the initial pseudo-labels (estimated correct answers) are biased or flawed.
Introducing COMPASS: A New Self-Scoring Reward Mechanism
A new framework called COMPASS (Composite Path and Answer Self-Scoring) has been introduced to overcome these limitations. COMPASS is a novel test-time reward mechanism that operates entirely without external supervision. It integrates two powerful components to enhance an LLM’s analytical capabilities by rewarding both trustworthy answers and high-quality reasoning processes.
Dual-Calibration Answer Reward (DCAR)
The first component is the Dual-Calibration Answer Reward (DCAR). This part focuses on creating more reliable pseudo-labels for stable learning. DCAR refines the traditional majority voting by incorporating two crucial signals: confidence and credibility. It hypothesizes that more confident responses should contribute more significantly to the final decision. The framework measures confidence using a metric called PD topk, which is the probability difference between the top-1 and top-2 tokens generated at each step. This metric strongly correlates with the correctness of the final answer, indicating the model’s predictive stability and decisiveness.
DCAR uses this confidence to weight responses in a process called confidence-calibrated self-consistency, where more decisive responses have a greater say in forming the consensus pseudo-label. Furthermore, it introduces a credibility score that assesses the quality of this consensus relative to the most confident individual response. This helps the model focus on highly credible pseudo-labels, leading to more stable and reliable optimization.
Decisive Path Reward (DPR)
While DCAR ensures robust outcome-level supervision, effective reasoning also requires guidance on the process itself. This is where the Decisive Path Reward (DPR) comes in. DPR provides dense, per-token feedback to optimize the reasoning trajectory. It encourages the LLM to make decisive actions at critical reasoning junctures, ensuring the integrity of the entire reasoning chain.
DPR evaluates two metrics at each generation step: decisiveness (how confident the model is in its token selection) and uncertainty (the entropy of the predictive distribution). The core idea is that decisiveness is more important during moments of high uncertainty. By dynamically weighting each step’s decisiveness by its corresponding uncertainty, DPR incentivizes confident actions when multiple alternatives seem viable, fostering more robust reasoning paths.
Also Read:
- Online Supervised Finetuning: A Simple Path to Enhanced LLM Reasoning
- Bridging Language and Numbers: How New AI Training Boosts LLM Reasoning
Impact and Future Direction
Extensive experiments have shown that COMPASS achieves significant and consistent performance gains across diverse reasoning tasks and model architectures, including benchmarks like AIME, AMC, MATH, and GPQA, using models like LLaMA and Qwen. The framework demonstrates natural scaling behavior, with larger models producing more accurate rewards and leading to more effective learning. It also promotes sustainable self-evolution by increasing pseudo-label accuracy while reducing over-reliance on a single popular answer, a common pitfall of naive majority voting.
COMPASS represents a significant step forward in enabling LLMs to learn autonomously from continuous experience streams, moving towards a future where models can improve themselves without constant human intervention. For more details, you can read the full research paper here.
