TLDR: A new training method called Online Supervised Finetuning (OSFT) significantly improves large language models’ (LLMs) reasoning abilities, particularly for mathematical tasks. Unlike complex reward-based reinforcement learning, OSFT is simple, reward-free, and highly efficient, using the model’s own self-generated responses for immediate finetuning. Experiments show OSFT achieves performance comparable to strong reinforcement learning methods, by enhancing the model’s existing knowledge and preferences through a decoupled temperature mechanism.
A new research paper introduces an innovative and surprisingly effective method for improving the reasoning capabilities of Large Language Models (LLMs), especially for complex tasks like mathematical problem-solving. This approach, called Online Supervised Finetuning (OSFT), stands out because it’s simple, doesn’t require external rewards, and is highly efficient.
Traditionally, enhancing LLM reasoning often involves complex reinforcement learning techniques that rely on verifiable rewards to guide the model’s learning. However, OSFT takes a different path. In this paradigm, the LLM essentially teaches itself: it generates its own responses to prompts and then immediately uses this self-generated data to finetune its parameters. This ‘self-help’ mechanism allows the model to iteratively refine its reasoning abilities without the need for a reward system.
One of the key advantages of OSFT is its efficiency. It’s ‘reward-free,’ meaning it avoids the computational overhead and complexity associated with designing and implementing reward functions. Furthermore, by default, it uses just one ‘rollout’ per prompt, which means it generates only a single sample response for each question during the data generation phase. This makes it a very lean and fast training strategy.
The researchers, Mengqi Li, Lei Zhao, Anthony Man-Cho So, Ruoyu Sun, and Xiao Li, demonstrated OSFT’s effectiveness through extensive experiments. They found that OSFT achieves performance on challenging mathematical reasoning tasks that is comparable to, and sometimes even surpasses, strong reinforcement learning with verifiable rewards (RLVR) methods like GRPO. This was observed across various mathematical benchmarks, including Math500, AMC, Minerva math, Olympiad-Bench, AIME24, and AIME25.
The core mechanism behind OSFT’s success lies in its ability to amplify the model’s existing preferences or ‘latent knowledge’ learned during its initial pretraining phase. The paper illustrates this with an example where a base model initially struggles with a math problem, often picking a slightly less probable but incorrect reasoning path. After OSFT training, the model learns to strongly prefer the correct reasoning path, significantly widening the probability margin between correct and incorrect steps. This isn’t about teaching the model entirely new facts, but rather aligning its generative process with the superior reasoning paths it already implicitly understands.
A crucial aspect of OSFT’s stability and effectiveness is the concept of ‘decoupled temperature dynamics.’ The researchers found that using different temperatures for sampling (generating data) and training (finetuning on that data) is essential. Specifically, a lower sampling temperature (Ï„s) combined with a standard training temperature (Ï„t = 1) leads to stable learning. If these temperatures are coupled (i.e., the same), the learning signal becomes directionless, and the model fails to improve.
The ablation studies conducted by the team further confirmed OSFT’s efficiency and robustness. They showed that even with a single rollout per prompt, OSFT performs strongly, and its performance is consistent across different base models (including specialized math models like Qwen2.5-Math-7B and general-purpose models like Qwen2.5-7B and Llama3.1-8B-Instruct) and different training datasets. This suggests that OSFT is a versatile and reliable method for improving LLM reasoning.
Also Read:
- Bridging Language and Numbers: How New AI Training Boosts LLM Reasoning
- Boosting LLM Performance with Strategy Fusion: Introducing SMaRT
In conclusion, Online Supervised Finetuning offers a compelling and efficient alternative to more complex, reward-based training paradigms for enhancing LLM reasoning. Its simplicity, reward-free nature, and comparable performance to state-of-the-art methods make it a promising direction for future research and application in the field of artificial intelligence. You can find the full research paper here.
