TLDR: New research reveals that while Large Language Models (LLMs) can achieve high math reasoning scores, only Reinforcement Learning (RL) fine-tuning effectively transfers these gains to broader tasks, unlike Supervised Fine-Tuning (SFT) which often causes models to ‘forget’ general abilities. RL preserves the model’s core understanding, leading to more versatile AI.
In the rapidly evolving field of large language models (LLMs), a critical question has emerged: does improving a model’s mathematical reasoning abilities also enhance its general problem-solving capabilities? A recent research paper, “Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning,” delves into this very question, offering surprising insights into how different training methods impact an LLM’s ability to transfer specialized skills to broader tasks. [1]
The authors, Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Poovendran, Graham Neubig, and Xiang Yue, investigated over 20 open-weight reasoning-tuned models across a diverse range of tasks, including math, scientific question answering, agent planning, coding, and standard instruction-following. Their findings challenge the intuitive assumption that strong performance in one complex domain, like mathematics, automatically translates to improved performance across the board. [1]
The Core Discovery: SFT vs. RL
The study’s most significant revelation is the stark difference in transferability between models fine-tuned using Supervised Fine-Tuning (SFT) and those using Reinforcement Learning (RL). While both methods can significantly boost a model’s math reasoning prowess, their impact on general capabilities varies dramatically. The researchers found that most models excelling in math through SFT often failed to transfer these gains to other domains, and in some cases, even experienced a decline in general abilities, a phenomenon known as “catastrophic forgetting.” [1]
In contrast, models fine-tuned with RL demonstrated remarkable generalization. Despite being trained solely on math-specific data, these RL-tuned models showed improved performance not only in other reasoning tasks but also in non-reasoning tasks like conversational question answering and instruction following. This suggests that RL helps LLMs learn in a way that preserves and even enhances their broader understanding and capabilities. [1]
Why the Difference? Latent Space and Token Distribution
To understand the underlying mechanisms behind this divergence, the researchers employed advanced diagnostic tools: latent-space Principal Component Analysis (PCA) and KL-divergence on token distributions. These analyses revealed that SFT induces substantial “drift” in the model’s internal representations and output space. This means that SFT, by focusing too narrowly on specific math patterns, can distort the model’s fundamental understanding, making it less effective for tasks outside its specialized training. [1]
Conversely, RL fine-tuning was found to preserve the general-domain structure of the model’s internal features and maintain the stability of its token distributions. This indicates that RL encourages a more targeted and efficient optimization process, where the model learns new skills without disrupting its existing knowledge base. RL models selectively adjust only task-relevant tokens, while SFT models tend to perturb many irrelevant tokens, leading to unnecessary “overthinking” in non-reasoning contexts. [1]
Also Read:
- Unlocking Advanced Math Skills in LLMs: The Power of Diversified Thinking
- Unlocking Deeper Reasoning: How ‘Modular Thinking’ is Revolutionizing LLMs
Implications for Future LLM Development
The findings of this research, published on July 1, 2025, have profound implications for how large language models are trained and refined. They suggest a critical need to re-evaluate standard post-training recipes, particularly the heavy reliance on SFT-distilled data for advancing reasoning models. The study highlights that while SFT can lead to impressive benchmark scores in specific areas, RL offers a more robust and generalizable approach to developing truly versatile and intelligent LLMs. [1]
This work underscores that achieving advanced reasoning capabilities in LLMs should not come at the cost of their general intelligence. By carefully selecting fine-tuning methodologies, particularly favoring reinforcement learning, developers can build models that not only excel in complex tasks like mathematics but also maintain and improve their broad problem-solving and understanding abilities across diverse domains. For more details, you can read the full research paper here. [1]
