TLDR: ACPO (Attribution-based Contribution to Policy Optimization) is a new framework for training Large Language Models (LLMs) using verifiable reinforcement learning. It solves key problems like inaccurate credit assignment (knowing which step was responsible for success/failure) and premature entropy collapse (LLM getting stuck in limited strategies). ACPO achieves this through dynamic segmentation of reasoning steps, a factorized reward system that precisely attributes credit to each step, and a two-stage curriculum that balances broad exploration with targeted refinement. Experiments show ACPO significantly improves LLM performance on complex math reasoning tasks.
Large Language Models (LLMs) are becoming increasingly vital for tackling complex cognitive tasks such as mathematical reasoning and multi-step decision-making. To enhance their capabilities, the field has seen a shift from traditional Supervised Fine-Tuning (SFT) to more advanced Reinforcement Learning (RL) frameworks. Among these, Reinforcement Learning with Verifiable Rewards (RLVR) stands out due to its ability to use objective, automated verifiers (like code compilers or math provers) for feedback, reducing annotation costs and subjective human biases.
However, current RLVR methods face significant hurdles. A major challenge is the difficulty in accurately assigning credit to intermediate steps within a long reasoning chain. When an LLM successfully solves a problem or makes an error, it’s hard to pinpoint exactly which specific steps contributed positively or negatively. This “credit assignment problem” leads to inefficient learning, where the model might penalize entire sequences without understanding the root cause of an error. Another critical issue is “premature entropy collapse,” where the model converges too quickly to a narrow set of reasoning strategies, limiting its ability to explore diverse and potentially better solutions.
Introducing ACPO: A Novel Framework for Enhanced RLVR
To address these fundamental challenges, researchers have introduced a new framework called Attribution-based Contribution to Policy Optimization (ACPO). ACPO is a two-stage algorithmic framework designed to significantly improve both the exploitation of known good strategies and the exploration of new ones in RLVR. You can read the full paper here: PINPOINTING CRUCIAL STEPS: ATTRIBUTION-BASED CREDITASSIGNMENT FORVERIFIABLEREINFORCE-MENTLEARNING.
Precise Credit Assignment for Better Exploitation
ACPO tackles the credit assignment problem with a sophisticated, factorized reward system. It uses a technique called trajectory semantic segmentation, which intelligently breaks down the LLM’s reasoning process into distinct, meaningful steps. Unlike rigid, rule-based segmentation, ACPO’s dynamic strategy identifies crucial decision points by focusing on “high-entropy tokens”—tokens that represent points of high uncertainty or important logical transitions. This allows the model to organically identify the boundaries of reasoning steps.
Once steps are identified, ACPO employs an attribution-based representation to quantify the hierarchical contribution of each reasoning step to the final outcome. This is done by measuring the “causal impact” of a step on the answer using a lightweight approximate attribution metric, essentially calculating how much new information a step brings to predicting the final answer. Steps that provide little new information are considered less effective.
This fine-grained understanding allows ACPO to assign “attribution advantage” to each step. For helpful steps, it encourages useful exploration by increasing rewards for high-entropy (low-confidence) outputs, allowing the model to try diverse yet effective paths. For low-entropy (high-confidence) helpful steps, the bonus is minimal, reinforcing known good strategies. Conversely, for harmful steps, ACPO reduces the entropy bonus, discouraging unproductive exploration and guiding the model towards more coherent sequences.
Dynamic Exploration Management with a Two-Stage Curriculum
To prevent premature entropy collapse and systematically manage the exploration-exploitation trade-off, ACPO introduces a progressive two-stage curriculum learning strategy:
- Stage 1: Broad Exploration: The initial phase focuses on maximizing exploration. It uses a KL-free objective, meaning it removes constraints on the policy, allowing the model to explore a wider range of solutions. Rewards are distributed uniformly within each reasoning step, and for more difficult problems, hierarchical sampling with a higher temperature is used to boost output diversity.
- Stage 2: Targeted Convergence: After broad exploration, the second stage shifts to refining the discovered strategies. A KL-divergence penalty is introduced to stabilize training and ensure the policy doesn’t stray too far from the effective strategies found in Stage 1. Crucially, reward allocation is refined by prioritizing “high-confidence tokens” within each step, encouraging the model to commit to its most certain and correct reasoning paths.
Experimental Validation and Promising Results
The ACPO algorithm was rigorously tested using the Qwen2.5-Math-7B model on challenging mathematical reasoning benchmarks, including AIME 2024, AIME 2025, AMC 2023, and MATH500. The results demonstrated that ACPO significantly outperforms existing state-of-the-art approaches, including the GRPO baseline, achieving an average improvement of 20% across various math evaluation sets. The experiments also showed that ACPO successfully maintains high entropy during early training stages, indicating effective exploration, and then reduces entropy in later stages as it identifies and optimizes critical reasoning steps. This leads to more robust and effective policies.
Also Read:
- CLARITY: Enhancing LLM Reasoning Quality Through Consistency-Aware Reinforcement Learning
- Sandwiched Policy Gradients: A Breakthrough in Diffusion Language Model Training
Conclusion
ACPO represents a significant advancement in verifiable reinforcement learning for LLMs. By providing a method for effective step classification using entropy, a mechanism for step-grained advantage attribution, and an exploration-centric two-stage training curriculum, ACPO resolves the long-standing credit assignment problem and optimizes the exploration-exploitation balance. Its ability to organically identify reasoning steps, rather than relying on rigid prompting, offers a more flexible and generalizable solution for complex reasoning tasks, paving the way for LLMs with enhanced reasoning capabilities.
