TLDR: A new algorithm, G2RPO-A, significantly improves the reasoning abilities of small-size language models (SLMs) by adaptively injecting ground-truth reasoning steps into their training. Unlike fixed guidance methods, G2RPO-A dynamically adjusts the amount of guidance based on the model’s learning progress and uses curriculum learning, leading to substantial performance gains in mathematical reasoning and code generation tasks.
In the rapidly evolving field of artificial intelligence, Large Language Models (LLMs) have shown incredible abilities, especially in complex reasoning tasks. A key technique behind this success is Reinforcement Learning with Verifiable Rewards (RLVR), which uses reinforcement learning to train LLMs based on rule-based outcomes. While highly effective for large models, this approach often yields only modest improvements for Small-size Language Models (SLMs) because they lack the extensive world knowledge of their larger counterparts.
A popular algorithm in RLVR is Group Relative Policy Optimization (GRPO). GRPO is an improvement over traditional methods like PPO because it doesn’t need a separate ‘critic’ model, which speeds up training. However, GRPO’s effectiveness is significantly limited when applied to SLMs. These smaller models struggle to generate high-quality, correct responses for complex problems, leading to a lack of strong feedback signals that are crucial for effective training.
To tackle this challenge, researchers explored a new approach called Guided GRPO. The core idea is to inject ‘ground-truth’ reasoning steps – essentially, correct thinking processes – directly into the model’s training attempts. This guidance is meant to help SLMs overcome their inherent weaknesses and produce better results. However, initial experiments revealed that simply adding a fixed amount of guidance, or ‘naive guidance,’ didn’t lead to significant improvements. It might offer a temporary boost but quickly became indistinguishable from the original GRPO.
This led to a deeper investigation into how guidance should be applied. The researchers found two critical insights: First, the ‘guidance ratio’ (the proportion of training attempts that receive guidance) matters. Code generation tasks and smaller models generally benefit from more guidance, while larger models and mathematical reasoning tasks perform better with less. Second, the ‘guidance length’ (how much guidance is provided) should not be fixed; it needs to change as the model learns. A predefined schedule for guidance length proved ineffective, highlighting the need for a more dynamic solution.
These findings motivated the development of G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance. This innovative algorithm automatically adjusts the guidance strength based on the model’s real-time learning progress. The key mechanism is simple yet effective: if the model’s performance (measured by average reward) improves, the guidance length is reduced, making the next set of training examples harder. If performance drops, the guidance length is increased, making the examples easier. This continuous adjustment ensures that the training difficulty always matches the model’s current capabilities.
Furthermore, G2RPO-A incorporates a ‘curriculum learning’ strategy. This means that training examples are sorted by difficulty, from easy to hard. By presenting problems in increasing order of complexity, the algorithm ensures that the model’s learning progress is consistent, preventing situations where sudden jumps in difficulty might skew the adaptive guidance mechanism. This approach also contrasts with methods that simply filter out very easy or very hard examples, which G2RPO-A avoids by retaining tough problems and providing adaptive guidance to help the model learn from them.
Experiments on mathematical reasoning and code generation benchmarks confirmed the effectiveness of G2RPO-A. It substantially outperformed both the vanilla GRPO and simple guided baselines across various models, including the Qwen3 series. The results showed clear performance gains, demonstrating that adaptively adjusting guidance is crucial for enhancing the reasoning abilities of small-size LLMs.
Also Read:
- Enhancing Language Model Reasoning Through Targeted Exploration
- Boosting LLM Reasoning: A New Approach to Self-Optimization with Entropy
This research marks a significant step forward in making advanced reasoning capabilities accessible to smaller language models, broadening the applicability of powerful AI techniques. For more details, you can read the full paper here.
