TLDR: This paper introduces Set Supervised Fine-Tuning (SSFT), a novel method to train large language models (LLMs) to reason in parallel. SSFT uses “global forking tokens” (e.g.,
Large Language Models (LLMs) have shown remarkable progress in reasoning, often by increasing the amount of computation during testing. However, simply generating more tokens sequentially can sometimes lead to “overthinking” and decreased performance. A more effective approach involves parallel sampling and aggregation, where the LLM generates multiple reasoning paths simultaneously and then combines them. The challenge here is ensuring these parallel paths are both diverse and accurate.
The core issue is that the specific “forking tokens” that trigger truly diverse and correct reasoning modes are often buried deep within the model’s sampling process. Traditional methods to encourage diversity, like adjusting temperature settings, often create a difficult trade-off between diversity and accuracy. This means that while you might get more varied responses, their correctness could suffer.
To address this, a new method called Set Supervised Fine-Tuning (SSFT) has been introduced. This approach treats parallel reasoning as a “set-of-next-token-prediction” problem. It incorporates a unique set-based global loss into the standard Supervised Fine-Tuning (SFT) process. The key innovation is the use of “global forking tokens” and a self-supervised bipartite matching technique.
Imagine you have a question, and you want the LLM to explore several different ways to solve it simultaneously. SSFT introduces special tokens, like <think1>, <think2>, etc., which act as prompts to initiate these distinct reasoning traces. The goal is for each of these global forking tokens to reliably trigger a unique and correct reasoning mode for a given question.
The bipartite matching process is crucial here. It’s like a sophisticated pairing system. For each question, the model has a set of global forking tokens and a set of diverse, correct reasoning traces (obtained from various “teacher” models or repeated sampling). The system then finds the best possible pairing between these tokens and traces, minimizing the “cost” (or prediction error) for each match. This ensures that a specific <think i> token is consistently associated with a particular type of reasoning trace, preventing different reasoning modes from collapsing into similar outputs.
SSFT works in two main steps during training: First, it calculates the compatibility (loss) between every global forking token and every reasoning trace. Then, it uses an efficient algorithm (like the Hungarian algorithm) to find the optimal pairings that minimize the total loss. Second, it updates the model’s parameters based on these optimal pairings, reinforcing the unique association between each forking token and its matched reasoning trace.
The benefits of SSFT are significant. Unlike naive fine-tuning with multiple reasoning traces, which often causes these unique modes to merge, SSFT preserves and even encourages the emergence of distinct reasoning patterns. This leads to improved performance on various reasoning benchmarks, measured by metrics like Pass@1 (accuracy of the single best answer) and Cons@k (accuracy when aggregating k parallel answers).
During inference, the learned global forking tokens are used to generate parallel responses. For example, for Cons@k, the model might be prompted with <think1>, <think2>, …, <thinkN> to generate N parallel reasoning paths, and then a majority vote determines the final answer. For Pass@1, the model selects the global forking token that has historically shown the broadest coverage or flexibility in reasoning.
Experiments have consistently shown that SSFT-trained models outperform standard SFT models, even when both are trained on the same diverse reasoning data. The optimal bipartite matching is key to this success, as models trained with random matching fail to achieve the same level of diversity and accuracy. Furthermore, SSFT demonstrates better coverage across different reasoning attempts and can genuinely initiate diverse reasoning traces, as evidenced by distinct reasoning lengths and performance profiles associated with different global forking tokens.
Also Read:
- MixReasoning: A Smart Approach to Efficient Language Model Thinking
- GUIDEDSAMPLING: Boosting LLM Performance Through Structured Exploration of Solution Concepts
This research highlights a powerful new way to train LLMs for parallel reasoning, making them more robust and capable of tackling complex problems by exploring multiple, distinct thought processes simultaneously. You can find the full research paper here.
