TLDR: This research paper investigates why reinforcement learning (RL) appears to boost the mathematical reasoning of Qwen2.5 models even with random or incorrect rewards, a phenomenon not observed in other models like Llama. The study reveals that Qwen2.5’s impressive performance on common benchmarks like MATH-500 is largely due to data contamination and memorization from its pre-training corpus. By introducing a new, clean dataset called RandomCalculation, the researchers demonstrate that only accurate reward signals lead to genuine and stable improvements in reasoning, while spurious rewards offer no benefit. This highlights the critical need for uncontaminated benchmarks to reliably evaluate LLM reasoning capabilities.
Recent advancements in large language models (LLMs) have sparked considerable excitement, particularly regarding their enhanced reasoning capabilities, often attributed to reinforcement learning (RL). Many new methods have emerged, claiming significant improvements with minimal or even no external supervision. Surprisingly, some studies even suggested that random or incorrect reward signals could boost reasoning performance, especially on the Qwen2.5 model family when evaluated on mathematical benchmarks like MATH-500, AMC, and AIME. However, these gains were often not replicated on other model families, such as Llama, prompting a deeper investigation.
A new research paper titled “REASONING OR MEMORIZATION ? UNRELIABLE RESULTS OF REINFORCEMENT LEARNING DUE TO DATA CONTAMINATION” by Mingqi Wu, Zhihao Zhang, Qiaole Dong, and their colleagues, delves into this puzzling discrepancy. Their empirical analysis suggests that while the Qwen2.5 series indeed shows superior mathematical reasoning, its extensive pre-training on massive web-scale data makes it susceptible to data contamination within these widely used benchmarks. This contamination, they argue, renders experimental results from Qwen2.5 on these benchmarks unreliable.
The Contamination Hypothesis
The researchers hypothesized that the Qwen2.5 model, trained on vast internet corpora, might have inadvertently absorbed parts of the evaluation benchmarks, including problems and their solutions, from public repositories like GitHub. If this were true, then seemingly random or incorrect rewards might simply be cueing the model to retrieve memorized answers rather than genuinely acquiring new reasoning skills.
To test this, they evaluated two key indicators: the model’s ability to complete a problem from a partial prompt and its accuracy in answering incomplete problems. Their findings were striking. When given only the first 60% of a MATH-500 question, Qwen2.5-Math-7B could accurately reconstruct over half of the remaining problem and still answer 53.6% of these incomplete problems correctly. In stark contrast, Llama3.1-8B showed significantly lower performance on both metrics.
Crucially, when tested on LiveMathBench (version 202505), a benchmark compiled *after* the release of the Qwen2.5 model family, Qwen’s completion rate plummeted to 0.0%, matching Llama’s performance. Its partial-prompt answer accuracy also dropped to a mere 2.0%, comparable to Llama’s 1.0%. These results strongly suggest that the earlier gains on MATH-500 were indeed due to memorized content, not genuine reasoning ability.
Introducing Clean Data: RandomCalculation
To provide a trustworthy evaluation signal, the researchers introduced a novel approach: a generator that creates fully synthetic arithmetic problems of arbitrary length and difficulty. This new, leakage-free dataset is called RandomCalculation. By ensuring that every problem in this dataset was created after Qwen2.5’s public release, they guaranteed no prior exposure or contamination.
Zero-shot evaluations on RandomCalculation confirmed the absence of memorization in Qwen2.5 models, with accuracy declining as the number of computation steps increased. This left ample room for improvement, indicating that the model wasn’t simply recalling answers.
The True Role of Rewards
Using these clean datasets, the researchers conducted controlled RL experiments. The outcome was unambiguous: only accurate reward signals led to consistent performance gains, pushing the model’s mathematical reasoning beyond its initial capabilities. Conversely, noisy or incorrect rewards resulted in highly unstable training and no reliable improvement. In fact, inverted rewards (where correct answers received zero reward and incorrect ones received one) rapidly eroded the model’s mathematical reasoning ability.
These findings directly refute the idea that Qwen’s strong baseline math skills alone explained its performance with spurious rewards on contaminated datasets. Once data leakage was removed, the previously observed “magical” gains vanished, underscoring the critical role of reward fidelity.
Also Read:
- Boosting Mathematical Reasoning in LLMs: A Two-Stage Training Strategy for Accuracy and Efficiency
- Unlocking LLM Reasoning with Minimal High-Quality Data
Implications for Future Research
The paper concludes by recommending that future studies evaluate LLMs on uncontaminated benchmarks and, whenever possible, test various model series to ensure trustworthy conclusions about RL and related methods. This work highlights that the impressive performance of some LLMs on mathematical reasoning tasks, particularly with seemingly arbitrary reward signals, might be a symptom of data contamination rather than a true advancement in reasoning. For more details, you can read the full research paper here.
