TLDR: The Compass-Thinker-7B model, developed by the Shopee LLM Team, demonstrates how Reinforcement Learning can be efficiently applied to smaller language models to achieve exceptional mathematical reasoning capabilities. By using a specially designed RL pipeline and a curated dataset of 30,000 math problems, the 7B-parameter model significantly improved performance on challenging benchmarks like AIME2024, proving that substantial AI advancements can be made with fewer computational resources.
Large Language Models, or LLMs, have made incredible strides in recent years, becoming fundamental to many applications like dialogue systems and code generation. A key technology in enhancing their capabilities, especially for complex reasoning, is Reinforcement Learning (RL). However, applying RL to massive models can be very expensive and resource-intensive, posing significant challenges.
Addressing this, the Shopee LLM Team has introduced the Compass-Thinker-7B model. This innovative model aims to explore the full potential of Reinforcement Learning using fewer computational resources and at a lower cost. It also provides valuable insights for future research into RL strategies for even larger AI models.
Compass-Thinker-7B is built upon an open-source model and trained using a specially designed Reinforcement Learning Pipeline. A crucial part of its development involved curating a unique dataset of 30,000 verifiable mathematics problems. By carefully configuring the data and training settings with different difficulty distributions across various stages, the model’s capabilities are gradually unlocked, leading to improved training efficiency.
Extensive evaluations have shown that Compass-Thinker-7B possesses exceptional reasoning potential. It achieves superior performance in mathematics compared to other RL models of the same size. Notably, in the challenging AIME2024 evaluation, Compass-Thinker-7B achieved an impressive 40% accuracy, a significant improvement over its base model’s 20%.
Reinforcement Learning from Human Feedback (RLHF) is a vital part of the LLM post-training process, primarily used to align models with human preferences. While effective, traditional RLHF methods are complex and resource-heavy. Researchers are continuously seeking simpler and more efficient algorithms. Compass-Thinker-7B leverages a variant of the Group Relative Policy Optimization (GRPO) algorithm, incorporating several industry improvements.
Instead of relying on complex reward models, Compass-Thinker-7B uses a rule-based reward system where the final accuracy of a verifiable task directly serves as the outcome reward. This straightforward approach has proven effective in activating the base model’s reasoning capabilities in various domains, including mathematics.
Key improvements integrated into Compass-Thinker-7B’s RL algorithm include removing the KL Loss (which measures deviation from the initial policy) to unleash more model potential without compromising stability. It also employs Dynamic Sampling, filtering out samples that don’t produce effective gradients, and a Clip-Higher strategy to prevent the model from losing diversity in its outputs during training.
The model was implemented using the verl framework, a flexible and efficient RL training library for LLMs. The base model for Compass-Thinker-7B was Qwen2.5-Math-7B. The training involved a batch size of 128, 16 gradient updates per iteration, and a learning rate of 1e-6, with a maximum sequence length of 4096 tokens to handle complex reasoning.
The dataset for RL training was collected from multiple open-source sources, such as NuminaMATH. Each problem had a verifiable ground truth, and the data underwent meticulous preprocessing, including deduplication, noise filtering, and difficulty grading. A custom verification program was developed to ensure accuracy and speed in filtering the training data.
Compass-Thinker-7B was rigorously evaluated on several challenging benchmarks, including the American Invitational Mathematics Examination 2024 (AIME2024), American Mathematics Competitions (AMC), Math 500, Minerva Math, and Olympiad Bench. The results consistently showed significant improvements across all benchmarks compared to the base model, and it outperformed other same-sized RL models in key evaluations.
Also Read:
- Klear-Reasoner: Enhancing AI Reasoning Through Refined Training Methods
- THINK TUNING: A New Approach to Teaching LLMs How to Self-Reflect
In conclusion, the Compass-Thinker-7B model demonstrates that it is possible to achieve highly competitive performance in mathematical reasoning through a meticulously designed Reinforcement Learning pipeline, even with fewer computational resources. This work, detailed in the Compass-Thinker-7B Technical Report, offers valuable insights for developing more powerful and efficient reasoning models in the future.
