TLDR: A study reveals that Reinforcement Learning (RL) compresses incorrect reasoning paths in LLMs, while Supervised Fine-Tuning (SFT) expands correct ones. At a granular level, RL concentrates reasoning functionality into fewer key steps, making the process more focused, whereas SFT distributes it across many steps, making it more diverse. This explains why combining SFT followed by RL is effective for enhancing LLM reasoning.
Large language models (LLMs) are increasingly used for complex logical tasks, especially in mathematics. To enhance their reasoning abilities, two primary training methods are commonly employed: Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) with verifiable rewards (RLVR). While both methods aim to improve performance, a recent study delves deeper into understanding how each method fundamentally shapes the reasoning process of these models, moving beyond simple accuracy measurements.
The research, titled “RL SQUEEZES, SFT EXPANDS: A COMPARATIVE STUDY OF REASONING LLMS,” by Kohsei Matsutani, Shota Takashiro, Gouki Minegishi, Takeshi Kojima, Yusuke Iwasawa, and Yutaka Matsuo from The University of Tokyo, introduces a novel framework to quantify and analyze reasoning paths. This framework examines reasoning at two distinct levels of granularity: the trajectory-level and the step-level. The study utilized models of varying sizes (1.5B, 7B, and 14B parameters) and focused on mathematical domains.
Understanding Reasoning Trajectories
At the trajectory-level, the researchers looked at complete reasoning outputs generated by the LLMs. They sampled multiple outputs from different model variants (Base, RL, SFT, and SFT+RL) and then clustered similar reasoning paths to identify unique trajectories. The findings revealed a complementary effect between RL and SFT.
Reinforcement Learning was observed to “squeeze” or compress incorrect reasoning trajectories. This means RL effectively reduces the number of unique wrong approaches a model might take. Conversely, Supervised Fine-Tuning was found to “expand” correct reasoning trajectories, increasing the diversity of unique and successful solution strategies. Interestingly, SFT alone tended to preserve incorrect trajectories, indicating it teaches new strategies but doesn’t necessarily prune the bad ones.
These observations provide a clear explanation for the success of the current best practice in LLM training: a two-stage approach where SFT is followed by RL. SFT first helps the model discover and expand a wider range of correct solution paths, and then RL steps in to suppress and eliminate the incorrect or inefficient paths, leading to maximized performance.
Analyzing Individual Reasoning Steps
The study then moved to a more granular step-level analysis, constructing “reasoning graphs” where each node represents an individual reasoning step (like a calculation or a logical expression). By analyzing the topology of these graphs, the researchers uncovered how RL and SFT modify the internal structure of reasoning.
RL was found to “steepen” the decay rates of node visitation frequency, degree, and betweenness centrality distributions in the reasoning graph. In simpler terms, RL concentrates the model’s reasoning functionality into a smaller subset of crucial steps. These steps become highly visited, connected, and central “hubs” in the reasoning process. This can be likened to a highly efficient, direct route where key decisions are made quickly.
In contrast, SFT “flattens” these decay rates, distributing reasoning functionality more homogeneously across many steps. This suggests that SFT encourages a more distributed and diverse set of reasoning steps, where importance is spread out rather than concentrated. While both RL and SFT introduce local cyclic structures (which can represent backtracking or verification steps), they lead to distinct global graph topologies. RL tends to transform the base model’s community-structured graphs into hub-centralized ones, while SFT, by learning from strong teacher models, creates more globally connected graphs with weaker community boundaries.
Also Read:
- Unlocking Dynamic Problem-Solving in AI with Explanatory Verifiers
- SIM-CoT: Enhancing LLM Efficiency and Accuracy Through Supervised Implicit Thoughts
Implications for LLM Training
The insights from this comparative study offer valuable guidance for developing more efficient learning approaches and data construction strategies for LLMs. By understanding that RL primarily refines and consolidates existing capabilities by pruning incorrect paths, and SFT primarily expands the model’s repertoire by teaching new correct strategies, developers can better design training pipelines. The paper suggests that applying RL specifically to functional or central reasoning steps could further boost performance. For more details, you can read the full research paper here.
