TLDR: This research paper introduces “Select-Then-Decompose” (S&D), an adaptive strategy for task decomposition in Large Language Models (LLMs) that balances performance and computational cost. Through empirical analysis, the authors identify a performance-cost dilemma in existing methods, noting that task characteristics and execution model scaling significantly influence outcomes. S&D uses a three-module closed-loop process (selection, execution, validation) to dynamically choose the best decomposition approach for a given task. Evaluations show S&D consistently achieves an optimal trade-off, favoring cost-efficient implicit methods and only escalating to more complex ones when necessary, thus enhancing efficiency and reliability.
Large Language Models (LLMs) have shown impressive abilities in reasoning and planning, leading to significant interest in how they break down complex tasks into smaller, manageable steps—a process known as task decomposition. While many existing methods focus on improving performance through memory, tool use, or feedback, they often overlook a crucial balance: the trade-off between achieving high performance and managing computational costs.
A recent study titled “Select-Then-Decompose: From Empirical Analysis to Adaptive Selection Strategy for Task Decomposition in Large Language Models” delves into this challenge. The researchers conducted a thorough investigation into task decomposition, identifying six different ways to categorize existing approaches. These categories include how decomposition and execution are interleaved, the number of LLM calls required, the structure of the decomposition (linear or directed acyclic graph), the format of subtasks (code or text), the range of subtask selection, and whether external tools are used during execution.
Understanding the Performance-Cost Dilemma
The study performed an in-depth analysis of three factors influencing task decomposition: the type of decomposition approach used, the characteristics of the task itself, and the configuration of the models involved in decomposition and execution. This analysis revealed a critical insight: current task decomposition methods face a significant performance-cost dilemma. Explicit approaches, which involve multiple LLM calls, often achieve superior performance but incur substantially higher costs in terms of token usage and API calls. For instance, some explicit methods consumed up to 10 times more tokens than implicit methods for certain tasks.
Another key finding was that task characteristics play a vital role in determining the most suitable decomposition strategy. For example, mathematical and reasoning tasks benefit from coherent, step-by-step reasoning methods like Chain of Thought (CoT) or Plan & Execute (P&E). Creative writing and comprehension tasks, which require more divergent thinking, align better with parallel decomposition structures like P&E (DAG). Code generation tasks, often requiring iterative refinement, showed better results with methods that alternate between reasoning and acting, such as ReAct.
The research also highlighted the impact of model discrepancies, particularly the scale and reasoning capabilities of the LLMs used. Scaling the execution model (the model that performs the subtasks) was found to yield greater performance gains than scaling the decomposition model (the model that breaks down the task). Additionally, models with enhanced reasoning capabilities significantly improved the execution stage.
Also Read:
- Boosting LLM Performance with Strategy Fusion: Introducing SMaRT
- Streamlining AI Reasoning: The Challenge of Deciding When to Think
Introducing Select-Then-Decompose (S&D)
Building on these insights, the researchers proposed a novel strategy called Select-Then-Decompose (S&D). This strategy aims to dynamically select the most appropriate decomposition approach based on the task’s complexity and characteristics, thereby achieving an optimal balance between performance and cost. S&D operates through a closed-loop problem-solving process with three main modules:
- Selection Module: An LLM analyzes the input question and, guided by specific prompts, determines the most suitable decomposition approach.
- Execution Module: This module applies the chosen decomposition approach to generate a candidate solution.
- Validation Module: An LLM assesses the confidence of the candidate solution. If the confidence is below a predefined threshold, the system initiates a staged switching mechanism, exploring more sophisticated approaches until a satisfactory solution is found or a maximum number of attempts is reached.
Extensive evaluations across various benchmarks demonstrated that S&D consistently operates on the Pareto frontier, meaning it achieves an optimal trade-off between performance and token cost. For tasks with smaller performance gaps, S&D achieved higher accuracy with minimal additional cost. For more complex tasks, it attained near-optimal performance while significantly reducing token consumption compared to other explicit methods.
The study also found that S&D primarily favors low-cost implicit strategies, only switching to more explicit or complex methods when tasks are particularly challenging or initial solutions fail validation. This adaptive behavior aligns with the practical principles derived from their empirical analysis, suggesting that matching the decomposition approach to task characteristics is key.
While promising, the S&D strategy has limitations. It currently focuses only on the decomposition mechanism and does not explore different representation formats (like code vs. text) or the use of external tools, which could further enhance performance for specialized tasks. Furthermore, its effectiveness relies on the strength of the LLM used in the selection module; a weaker model might lead to suboptimal choices. The full research paper can be found here: Select-Then-Decompose Research Paper.
This research provides valuable insights into optimizing task decomposition in LLMs, offering a practical framework for balancing performance and computational costs in real-world applications.
