TLDR: A new study introduces iSELF-DISCOVER, an instance-level adaptation of the SELF-DISCOVER framework, to compare structured JSON reasoning with unstructured natural language reasoning in LLMs. The research, using state-of-the-art open-source models across diverse benchmarks, consistently finds that unstructured reasoning offers a significant performance advantage, especially on complex tasks like MATH. It also reveals that the optimal granularity of reasoning plans (instance-level vs. task-level) and the utility of few-shot guidance are context-dependent, challenging the widespread reliance on structured outputs for LLM reasoning.
A new research paper titled “Effects of structure on reasoning in instance-level SELF-DISCOVER” by Sachith Gunasekara and Yasiru Ratnayake from Surge Global explores a fascinating aspect of how Large Language Models (LLMs) approach complex problem-solving. The study delves into whether structured reasoning, often seen in formats like JSON, or unstructured natural language reasoning is more effective for LLMs.
In the world of LLMs, there’s a growing need for predictable and controllable outputs, especially when these models are integrated into larger systems that might use tools or function calls. This has led to the popularity of structured output formats, with JSON being a prime example. However, a key question remains: do these structured formats hinder an LLM’s performance compared to its ability to reason freely in natural language?
The researchers introduce a new framework called iSELF-DISCOVER, which is an adaptation of the existing SELF-DISCOVER framework. While the original SELF-DISCOVER generates a single, task-level reasoning structure, iSELF-DISCOVER takes a more granular approach, creating a customized reasoning plan for each individual problem instance. This allows for a direct comparison between structured (dynamic JSON) and unstructured (natural language) reasoning at the same detailed level.
The study’s empirical evaluation, conducted across various benchmarks including BIG-Bench Hard (BBH), Thinking for Doing (T4D), and the complex MATH benchmark, used state-of-the-art open-source models like LLaMA-3.1-405B-Instruct and Mistral-Large. The findings consistently point to a significant advantage for unstructured reasoning. For instance, on the challenging MATH benchmark, unstructured plans showed a performance improvement of up to 18.90% over structured approaches when using LLaMA, and a 13.33% gain with Mistral.
Interestingly, the research also found that zero-shot unstructured iSELF-DISCOVER variants often outperformed their five-shot structured counterparts. This highlights the substantial gap in performance, even when structured plans are dynamically generated to ensure the reasoning process occurs before the final answer is given. This suggests that the free-form nature of natural language might align more closely with how these models are pre-trained.
The paper also sheds light on the optimal granularity of plan generation. For diverse benchmarks like BBH, instance-level reasoning (iSELF-DISCOVER) showed benefits, particularly with Mistral. However, for more coherent tasks like T4D, the task-level SELF-DISCOVER baseline demonstrated stronger performance. This indicates that there isn’t a one-size-fits-all solution; the best strategy depends on the characteristics of the task and the specific language model being used.
Furthermore, the influence of providing few-shot examples during instance-level plan generation was explored. While 5-shot guidance proved beneficial for tasks with high internal consistency, like T4D, its effect was marginal or even negative on highly diverse benchmarks suching BBH. This implies that for truly instance-adaptive approaches, a zero-shot configuration might be sufficient or even preferable, as a single instance can provide all necessary context without the risk of conflicting information from other examples.
Also Read:
- Large Language Models Show Human-Like Logic Induction, Challenging Cognitive Theories
- Boosting LLM Reasoning: A New Approach to Efficient Reinforcement Learning
In conclusion, this research invites a re-evaluation of the heavy reliance on structured formats for complex problem-solving in LLMs. It demonstrates the significant performance benefits of unstructured natural language reasoning, even within instance-level self-discovery processes. The findings also emphasize that choices regarding plan granularity and the use of few-shot guidance are context-dependent, varying with task characteristics. For more details, you can read the full research paper here.
