TLDR: A new method called SSPO (Self-traced Step-wise Preference Optimization) has been developed to improve Large Language Models (LLMs) by making their reasoning processes more accurate and concise. Unlike previous methods that can lead to ‘overthinking’ and require extensive resources or manual input, SSPO uses the LLM’s own internal signals to guide step-by-step optimization. This approach eliminates the need for extra models or human annotations, resulting in LLMs that provide accurate answers with significantly shorter and more efficient reasoning sequences across various tasks and languages.
Large Language Models (LLMs) have become incredibly powerful, but enhancing their performance after initial training often comes with significant computational costs. Traditional methods, particularly those involving reinforcement learning (RL) with chain-of-thought (CoT) reasoning, can lead to what researchers call ‘overthinking’ – where LLMs generate excessively long and sometimes error-prone reasoning sequences, even for simple tasks.
A new research paper introduces a novel framework called Self-traced Step-wise Preference Optimization (SSPO) that aims to make LLM reasoning both accurate and succinct. The core idea behind SSPO is to optimize each step of the LLM’s reasoning process in a fine-grained manner, without needing additional complex models or laborious manual annotations for each step.
The Problem with Current Methods
Existing RL-based methods like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) often rely on sparse reward signals, meaning they only evaluate the final answer’s correctness. This can incentivize LLMs to explore overly elaborate reasoning chains, leading to inefficiency and potential error accumulation. Other process supervision techniques, such as Monte Carlo Tree Search (MCTS) or Process Reward Models (PRMs), offer dense supervision but typically demand substantial computational resources or extensive human labeling.
Introducing SSPO: A Smarter Approach
SSPO addresses these limitations by leveraging the LLM’s inherent capabilities. It proposes a method called Verbal Value Probing (VVP), which allows the policy model itself to evaluate the value of each reasoning step. Instead of external models, VVP uses a clever technique: it adds a conclusion format to the end of each reasoning step, prompting the LLM to self-assess its current state and estimate the probability of reaching the correct answer. This provides a computationally efficient way to get dense, step-wise preference signals.
The researchers found that in cases where CoT reasoning led to errors (dubbed ‘CoT-poisonous’ queries), the step-wise value estimated by VVP showed a declining or fluctuating trend. This observation supports their hypothesis that overthinking stems from error-prone steps, which can be mitigated by providing dense, step-wise supervision that penalizes suboptimal reasoning pathways.
How SSPO Works
SSPO integrates these VVP-derived step-wise values into a reformulated advantage computation, similar to the Generalized Advantage Estimator (GAE) used in PPO. This dynamic calibration ensures that gradient updates prioritize logically consistent reasoning steps and discourage redundant explorations. Furthermore, SSPO includes an ‘Error Step Pruning’ strategy. This mechanism identifies points where the step-wise value begins to decline and prunes subsequent reasoning steps from contributing to gradient updates, effectively preventing the propagation of errors and overthinking.
Also Read:
- Adaptive Guidance Boosts Reasoning in Small Language Models
- Boosting LLM Logical Reasoning with Structured Chain-of-Thought
Impressive Results
Experiments conducted across various tasks and languages (mathematical reasoning and medical question-answering in English and Chinese) demonstrated SSPO’s effectiveness. When applied to baseline methods like GRPO and DAPO, SSPO consistently achieved comparable or even improved accuracy while significantly compressing the reasoning process. For instance, SSPO improved GRPO’s reasoning performance on a 7B LLM by 1.46% while reducing the reasoning process length by 36.91%. Notably, SSPO-trained LLMs automatically adapted to the appropriate reasoning length for different tasks, eliminating the need for manual length specifications.
The study also analyzed the entropy trajectory during training, showing that SSPO’s dense supervisory signals effectively constrain the model’s reasoning trajectory, balancing exploration and exploitation to prevent erroneous ‘aha moments’ that lead to overthinking. This leads to more stable and accurate reasoning pathways.
In conclusion, SSPO offers an efficient and pluggable solution for providing dense process supervision in rule-based RL methods for LLMs. By enabling LLMs to self-trace and optimize their reasoning step-by-step, SSPO effectively mitigates the overthinking problem, leading to more accurate and concise responses. You can read the full paper here.
