TLDR: A new research paper introduces ExpandSearch, a reinforcement learning framework that trains LLMs to improve search accuracy by generating multiple diverse queries (expand phase) and then using a ‘squeezer’ model to distill relevant information from the search results (squeeze phase). This ‘expand-then-squeeze’ strategy significantly enhances performance on complex multi-hop question-answering tasks, allowing even smaller LLMs to achieve state-of-the-art results by overcoming limitations of single queries and information overload.
Large Language Models (LLMs) have become incredibly powerful, especially when combined with search capabilities to answer complex questions. These ‘reasoning-augmented search agents’ can search for information and refine their answers iteratively. However, they often struggle with complex questions that require multiple steps of reasoning, mainly due to two key issues: their queries are often too narrow, missing crucial information, and they can get overwhelmed by too much irrelevant information from search results.
A new research paper, titled “Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learning,” introduces an innovative approach called ExpandSearch to tackle these challenges. Authored by Shu Zhao, Tan Yu, and Anbang Xu, this work proposes a novel framework that significantly enhances how LLMs interact with external knowledge sources.
The Expand-then-Squeeze Strategy
The core idea behind ExpandSearch is a two-phase strategy: ‘expand’ and ‘squeeze.’ This mirrors how humans often search for information – first casting a wide net, then carefully filtering what’s found.
In the expand phase, instead of generating a single query, the LLM is trained to propose several diverse query variants simultaneously. These variants can include paraphrases, decomposed sub-questions, or keyword expansions. This multi-query approach helps cover a broader range of relevant information, overcoming the limitations of a single, potentially narrow, search query. The paper highlights two types of expansions: syntax expansions, which handle different ways of phrasing the same idea (e.g., “where did he die” vs. “his death place”), and semantic expansions, which broaden the scope to related concepts (e.g., “Alexander’s father” vs. “Alexander’s family”).
The squeeze phase addresses the problem of information overload. When multiple queries are run, a large amount of information is retrieved, much of which might be irrelevant. To prevent the LLM from getting distracted, a separate ‘squeezer’ model is used. This squeezer, a pre-trained LLM, distills the lengthy retrieved content into a compact summary, focusing only on the facts critical for answering the question. This significantly reduces the computational burden and helps the main search agent concentrate on generating effective queries.
Training and Performance
ExpandSearch trains the LLM-based search agent using reinforcement learning. This means the agent learns through trial and error, optimizing its query generation and information utilization strategies based on rewards for correct answers and proper formatting. The researchers found that this end-to-end training is crucial; simply adding query expansion and a squeezer without this specialized training actually degraded performance, showing that the model needs to learn how to effectively use these new capabilities.
The experiments, conducted across seven different question-answering benchmarks, demonstrated impressive results. ExpandSearch achieved an average improvement of 4.4% compared to state-of-the-art baselines. Notably, even a smaller 3-billion parameter LLM trained with ExpandSearch could outperform larger 7-billion parameter models from other methods, indicating the efficiency and effectiveness of the expand-then-squeeze approach.
The method showed strong gains particularly in ‘multi-hop’ reasoning tasks, which require piecing together information from multiple sources. This is where the expanded query coverage and distilled context proved most beneficial. The analysis also revealed that syntax expansions were more frequently generated by the trained model, suggesting that variations in phrasing pose a significant challenge for current embedding-based retrieval systems.
Also Read:
- Enhancing Language Model Reasoning Through Representation-Based Exploration
- Dynamic Temperature Control Enhances LLM Reasoning in Reinforcement Learning
Future Implications
This research suggests that the future of advanced search agents lies not just in making LLMs larger or retrieving more documents, but in teaching them smarter strategies for formulating queries and synthesizing information. The modular design of ExpandSearch also allows for flexibility, enabling different squeezer models to be used based on task requirements or computational resources without needing to retrain the entire system. For more details, you can read the full paper here.
