TLDR: The paper introduces “off-trajectory reasoning” to evaluate if LLMs can effectively collaborate by assessing and building on partial reasoning from others. It proposes Recoverability (backtracking from distractions) and Guidability (building on correct guidance) tests. Findings show that current LLMs, even strong solo reasoners, perform poorly in these collaborative scenarios, often failing to recover from distractions or leverage useful hints. Control studies reveal that teacher model vulnerabilities transfer to students, RL can improve these behaviors, and limited high-quality data can lead to high variance in off-trajectory performance.
Large Language Models (LLMs) have made incredible strides in complex tasks like mathematics and coding, largely due to their ability to verbalize their thought processes. This transparency has opened up an exciting new frontier: the possibility of multiple LLMs, or even humans, collaborating directly on a shared reasoning path. This concept, known as “off-trajectory reasoning,” explores whether these models can assess and build upon each other’s partial thinking, leading to more efficient and exploratory problem-solving.
However, a critical question arises: are current LLMs, primarily trained for “solo-reasoning,” equipped for this collaborative challenge? A recent research paper, “OFF-TRAJECTORY REASONING: CAN LLMS COLLABORATE ON REASONING TRAJECTORY?” by Aochong Oliver Li and Tanya Goyal from Cornell University, delves into this very question. Their work introduces a novel framework to evaluate LLMs’ ability to engage in off-trajectory reasoning, revealing some counterintuitive findings about their collaborative capabilities.
Understanding Off-Trajectory Reasoning
The researchers define off-trajectory reasoning as an LLM’s capacity to integrate useful insights from collaborators and reliably backtrack from incorrect or unhelpful inputs, even if these inputs are outside their usual training distribution. To systematically evaluate this, they propose two complementary tests: Recoverability and Guidability.
The Recoverability Test
The Recoverability test assesses an LLM’s robustness in backtracking from erroneous reasoning introduced by a collaborator. Imagine an LLM is solving a math problem, and suddenly, a “distracting steer” – a snippet of reasoning from a different, unrelated problem – is inserted into its thought process. The test measures if the LLM can ignore this distraction and return to its original, correct reasoning path to find the right answer. Surprisingly, the study found that “stronger” LLMs, those that perform well on standard benchmarks, are often more fragile under such distractions, with performance drops observed across models.
The Guidability Test
On the other end of the spectrum, the Guidability test evaluates an LLM’s ability to build upon correct, yet incomplete, reasoning provided by a “guiding steer” from a stronger collaborator. This test focuses on problems that the LLM struggles to solve on its own. The goal is to see if the model can leverage these hints to surpass its inherent solo-reasoning limitations. The results here were even more sobering: all models tested failed to effectively use guiding steps from collaborators, with solve rates remaining very low, even when the guiding steer contained the correct answer.
Key Findings from the Evaluation
The study evaluated 15 open-weight LLMs (ranging from 1.5B to 32B parameters) across five math benchmarks. The main takeaways were:
- Strong Solo-Reasoners Are Not Necessarily Strong Collaborators: Models that excel in solo-reasoning benchmarks often showed poor recoverability and guidability. For instance, the best-performing solo-reasoning model exhibited one of the worst recoverability performances.
- The Beginning of Reasoning is Critical: Distractions introduced at the very start of an LLM’s reasoning trajectory led to the largest performance degradation. This suggests that the initial restatement of a question, while seemingly trivial, is crucial for anchoring the model’s subsequent reasoning.
- Failure to Leverage Guidance: LLMs consistently struggled to build on correct guidance, even when the guiding steer contained the correct answer derivation. This indicates a fundamental limitation in integrating out-of-distribution but useful information.
Insights from Control Studies on Post-Training
To understand what factors influence these off-trajectory behaviors, the researchers conducted control studies focusing on three aspects of post-training:
- Teacher Model Behavior: They found that the recoverability vulnerabilities of a teacher model used for distillation directly transfer to its student models, even if the distillation data itself is correct. This highlights the importance of considering a teacher’s collaborative behaviors, not just its correctness.
- Reinforcement Learning (RL): RL training, particularly with outcome-based rewards, showed significant improvements in both recoverability and guidability, even after Supervised Fine-Tuning (SFT) had plateaued. This suggests RL can expose models to noisy trajectories and explicitly reward successful recoveries.
- Data Selection Strategy: Models trained on smaller, highly filtered “high-quality” datasets exhibited much higher variance in recoverability across different training checkpoints compared to models trained on larger, more diverse datasets. This implies that aggressive data filtering, while boosting benchmark scores, might introduce biases that negatively impact robustness in off-trajectory scenarios.
Also Read:
- Task Complexity: A Key to Effective LLM Multi-Agent Systems
- Unpacking LLM Memory: New Research Reveals Early Forgetting in Complex Reasoning Tasks
Conclusion
This research lays crucial groundwork for evaluating multi-model collaborations in shared reasoning trajectories. It highlights that current off-the-shelf reasoning LLMs have significant limitations when it comes to collaborative thinking, struggling to both resist distractions and effectively leverage external guidance. The findings provide actionable insights for training future LLMs to be natively strong reasoning collaborators, emphasizing the need to consider off-trajectory behaviors alongside traditional benchmark performance.
