TLDR: Adaptive Passage Combination Retrieval (AdaPCR) is a new framework for Retrieval-Augmented Generation (RAG) that improves open-domain question answering, especially for multi-hop questions. Unlike traditional RAG that retrieves passages independently, AdaPCR explicitly models dependencies between passages by considering passage combinations as units for retrieval and re-ranking. It uses a two-stage process: initial dense retrieval followed by combination re-ranking with context-aware query reformulation. Trained with a predictive objective aligned with answer likelihood, AdaPCR adaptively selects the optimal number of passages, outperforming baselines on QA benchmarks by reducing redundancy and enhancing diversity.
Large Language Models (LLMs) have become incredibly powerful, but their knowledge can quickly become outdated, and updating them through retraining is very expensive. This is where Retrieval-Augmented Generation (RAG) comes in. RAG enhances LLMs by allowing them to access external, up-to-date information during inference, without the need for costly retraining. It’s a simple yet effective way to keep LLMs informed.
However, traditional RAG methods often retrieve information in a way that has some significant drawbacks. They typically fetch passages independently, which can lead to several issues. Imagine searching for information and getting a lot of repetitive or irrelevant results – that’s what can happen. This redundancy wastes valuable space in the LLM’s context window and can prevent the model from getting all the necessary information. This problem is even more pronounced when dealing with noisy datasets or complex questions that require combining information from multiple sources, known as multi-hop questions. Furthermore, existing methods often retrieve a fixed number of passages, which might be too many (introducing noise) or too few (missing crucial details).
To tackle these challenges, researchers have introduced a new framework called Adaptive Passage Combination Retrieval (AdaPCR). This innovative approach is designed for open-domain question answering, especially when working with ‘black-box’ LLMs (meaning you can use them but can’t easily fine-tune their internal workings). AdaPCR moves beyond retrieving passages one by one; instead, it explicitly considers combinations of passages as a single unit for retrieval and re-ranking. This allows the system to understand the relationships and dependencies between different pieces of information.
How AdaPCR Works
AdaPCR operates in two main stages. First, it performs an initial retrieval to gather a small pool of top candidate passages based on their relevance to the question. Think of this as a preliminary search to narrow down the possibilities. Second, and this is where AdaPCR truly shines, it enters a ‘combination re-ranking’ stage. For each passage retrieved in the first stage, it’s combined with the original question to form a new, context-aware query. This new query is then used to find additional relevant passages, creating potential combinations. By doing this, AdaPCR can score and select the best combination of passages, which might include one or two passages, that are most likely to help the LLM generate an accurate answer. This process implicitly balances the need for relevance with the need for diversity in the retrieved information, ensuring the LLM gets the most informative context.
Training the System
To make AdaPCR effective, its re-ranker is trained using a ‘predictive re-ranking’ objective. This means the system learns to align its scoring of passages with how well the LLM can actually answer the question using those passages. A key aspect of this training is the use of RAG loss, which has been shown to be particularly effective. This loss function helps the system handle varying answer likelihoods and provides a stable learning signal, encouraging the re-ranker to prioritize passages that lead to higher answer quality from the LLM.
Performance and Results
Experiments were conducted on two well-known question-answering datasets: Natural Questions (for single-hop questions) and HotpotQA (for multi-hop questions). The results demonstrated that AdaPCR consistently outperformed existing baseline methods, especially when dealing with complex multi-hop reasoning tasks. For instance, on HotpotQA, AdaPCR showed significant improvements in both Exact Match (EM) and F1 scores, indicating its superior ability to handle questions requiring information from multiple sources. The research also highlighted that the balanced sampling strategy used during training significantly improved stability and convergence, particularly for multi-hop questions.
A notable finding was AdaPCR’s effectiveness in handling both single-hop and multi-hop questions, with particularly strong gains in complex, reasoning-intensive settings. This flexibility in dynamically selecting passages and aligning with downstream objectives contributes to its robustness across various QA tasks. Even general-purpose language models like BERT, without specific fine-tuning for retrieval, showed substantial enhancement in their retrieval capabilities when integrated into the AdaPCR framework.
Also Read:
- SARA: Enhancing RAG Performance Through Hybrid Context Management
- Advancing RAG Through Progressive Instruction Tuning
Real-World Examples
The paper provides compelling case studies to illustrate AdaPCR’s advantages. In one example, for the question “Which band is from England, Fireflight or Dirty Pretty Things?”, traditional methods often retrieved redundant or irrelevant information. In contrast, AdaPCR successfully retrieved complementary evidence: one passage confirming facts about “Dirty Pretty Things” and another establishing “Fireflight” as an American band. This allowed for a clear and correct answer. Another case study on a multi-hop question about the “Argentine National Anthem” showcased AdaPCR’s ability to leverage information from one passage to reformulate the query, leading to the retrieval of a second crucial passage that provided the missing piece of the puzzle. This demonstrates the framework’s strength in synthesizing information from multiple sources for accurate reasoning.
In conclusion, Adaptive Passage Combination Retrieval (AdaPCR) offers a promising advancement in RAG systems. By explicitly modeling dependencies between passages and adaptively selecting combinations, it addresses critical limitations of conventional approaches, leading to more accurate and robust open-domain question answering, particularly for complex queries. You can read the full research paper here.
