TLDR: The AdaR framework improves Large Language Models’ (LLMs) mathematical reasoning by addressing “spurious reasoning.” It synthesizes diverse, logically equivalent math problems by perturbing variable values and extracting problem-solving logic as code. Then, it trains LLMs using Reinforcement Learning with Verifiable Rewards (RLVR) on this data, penalizing superficial logic and promoting adaptive, logic-based reasoning. AdaR significantly boosts robustness and generalization in LLMs with minimal data.
Large Language Models, or LLMs, have shown impressive capabilities in many areas, but mathematical reasoning remains a significant challenge. Researchers have observed that these models often fail in terms of robustness and generalization when tackling math problems. This isn’t because they can’t get the right answer sometimes, but because they often arrive at it through “spurious reasoning” – essentially, guessing based on superficial patterns rather than truly understanding the underlying logic.
To address this, a new framework called AdaR has been introduced. AdaR aims to foster “adaptive reasoning” in LLMs, meaning the models learn to rely on the actual problem-solving logic. Imagine a student who can solve a math problem by memorizing the answer for specific numbers, versus a student who understands the formula and can solve it for any numbers. AdaR wants LLMs to be the latter.
The AdaR framework has two main parts: data synthesis and model training.
Data Synthesis: Creating Smart Practice Problems
The first step involves creating a diverse set of training data. Instead of just giving the model more problems, AdaR generates new, logically equivalent versions of existing problems. This is done by taking an original math problem, extracting its core problem-solving logic (often represented as executable code, like a Python program), and then changing the numerical values in the problem. For example, if the original problem asks to find ‘A’ where ’32 × A × A × A = 42592′, AdaR might create a variant like ’35 × A × A × A = 53550′ while keeping the underlying cubic root logic the same.
This process is carefully controlled. The framework ensures that the new problems are “sane” – meaning they have valid solutions and the variables align correctly with the problem-solving code. It also uses “sanity checks” to filter out any invalid instances that might arise from the perturbations. This ensures that the synthetic data is high-quality and truly helps the model learn the logic, not just new superficial patterns. Query paraphrasing is also used to increase the diversity of problem templates, making the model more adaptable to different phrasing of the same problem type.
Model Training: Learning from Comparison
Once the synthetic data is generated, AdaR trains the LLM using a technique called Reinforcement Learning with Verifiable Rewards (RLVR). Traditional RLVR often just checks if the final answer is correct. However, AdaR uses its unique synthetic data to provide more nuanced feedback. When an LLM is given a set of perturbed queries (all based on the same underlying logic but with different numbers), AdaR can observe if the model consistently gets the right answers across all variants. If the model relies on spurious reasoning, it’s more likely to fail on the perturbed versions, leading to a penalty. If it uses adaptive reasoning, it will succeed, and its adaptive logic will be reinforced.
This comparative feedback mechanism is crucial. It pushes the model to explore and adopt the correct problem-solving logic, rather than just memorizing specific solutions or superficial features. The paper highlights that this coordinated approach of data synthesis and RLVR is vital for enabling adaptive reasoning.
Also Read:
- FormalML: A New Benchmark to Bridge AI and Advanced Mathematical Proofs
- Language Models Struggle with False Premises: Insights from the BROKENMATH Benchmark
Impressive Results and Broad Applicability
Experiments showed that AdaR significantly boosts LLMs’ performance in mathematical reasoning. Using a relatively small amount of synthetic data (just 9,000 instances), AdaR achieved substantial improvements in both robustness (how well it performs on similar, slightly altered problems) and generalization (how well it performs on entirely new types of problems). For instance, it showed an average gain of 8.50 points over other methods.
The framework also demonstrated that its effectiveness is linked to the base model’s initial mathematical reasoning ability, suggesting it complements existing mathematical pre-training. Furthermore, AdaR was found to be applicable to “Instruct” models, which are already highly tuned, providing an additional performance boost even for these advanced models.
Analysis revealed that AdaR helps LLMs develop “algebraic thinking,” where they treat unknown and known variables equally and solve problems through variable calculation. It also enhances the model’s adherence to logical order, a key characteristic of adaptive reasoning. The research also explored factors like the magnitude of numerical perturbations, finding an optimal balance to maximize learning without introducing too much noise.
In conclusion, AdaR offers a powerful and scalable framework for improving mathematical reasoning in LLMs. By synthesizing high-quality, diverse data and leveraging RLVR for adaptive learning, it helps models move beyond superficial pattern matching to truly understand and apply problem-solving logic. You can find the full research paper here.
