TLDR: The R2U framework introduces a novel approach to improve Retrieval-Augmented Generation (RAG) systems by directly optimizing document utility for answer generation. It uses “process supervision” from large language models to train smaller rewriter models, ensuring retrieved documents are not just relevant but truly useful for generating accurate answers, especially for complex queries. This method consistently outperforms existing baselines across various question-answering tasks and generalizes well to different generator models.
Retrieval-Augmented Generation (RAG) systems have become a cornerstone for enabling large language models (LLMs) to answer questions based on information not seen during their initial training. By fetching relevant documents and then using a generator to synthesize a response, RAG aims to provide more accurate and up-to-date answers. However, a persistent challenge in RAG systems is the disconnect between document relevance and generative utility. Often, documents retrieved might be topically relevant but still lack the specific content or structure needed for the LLM to perform effective reasoning and generate a correct answer.
Existing methods, often referred to as “bridge” modules, attempt to address this by rewriting or reranking retrieved text to better support generation. These include techniques like reranking documents based on their perceived usefulness or summarizing them to focus on query-relevant information. While these approaches offer some improvements, a new research paper titled Relevance to Utility: Process-Supervised Rewrite for RAG highlights that these methods often fall short in truly capturing what makes a document useful for generating an accurate answer.
The paper introduces a novel framework called R2U, which stands for Relevance to Utility. The core distinction of R2U is its direct optimization to maximize the probability of generating a correct answer, achieved through a technique called “process supervision.” This means R2U doesn’t just look at whether a document is relevant, but whether it actively helps the LLM produce the right answer by observing the LLM’s internal reasoning process.
Understanding Process Supervision
Directly observing how a document contributes to a correct answer can be computationally expensive, especially with very large LLMs. To overcome this, R2U proposes an efficient distillation pipeline. It leverages the reasoning traces of robust LLMs (like those with 70 billion parameters or more) as a “teacher” to train a smaller rewriter model. These large LLMs, during their internal reasoning, naturally perform the desired rewriting, producing modified documents that are specifically tailored to help generate a target answer. This internal process is what R2U captures as process supervision.
The smaller rewriter model then learns to imitate this behavior. To make this distillation more robust and help the smaller model generalize better, R2U employs “inference scaling.” This involves augmenting the training data by having the LLM rewrite individual documents, often placing the target document at the end of the context to mitigate common positional biases that smaller models might exhibit.
Training the R2U Rewriter
The training of the smaller R2U rewriter model involves two main stages. First, it undergoes supervised fine-tuning, where it learns to generate rewritten document sets based on the original retrieved documents and the query, imitating the teacher LLM’s process. Second, the model is further optimized using Direct Preference Optimization (DPO). DPO helps the smaller model adapt its rewriting preferences to what truly leads to correct answers, rather than just strictly mimicking the teacher. This is particularly innovative because it constructs preference pairs not just from individual documents, but from sets of documents with varying degrees of individual answer correctness, allowing for a more nuanced understanding of utility.
Also Read:
- Boosting Information Retrieval with Chunk-Based Knowledge Generation
- Advancing Conversational Search with Intelligent Query Reformulation and Result Merging
Empirical Results and Impact
The R2U method was evaluated across multiple open-domain question-answering benchmarks, including multi-hop QA datasets like HotpotQA, 2WikiMultihopQA, and MuSiQue, as well as datasets requiring disambiguation like AmbigQA. It also demonstrated strong performance on web corpus benchmarks like MS MARCO and CRAG. The empirical results consistently show significant improvements over strong existing baselines.
For instance, R2U improved the average F1 score by 8.8% points over a naive RAG baseline and outperformed the best existing baselines by 7.8% points. These gains were particularly pronounced in scenarios demanding higher document utility, such as aggregation, multi-hop, and post-processing-heavy queries, where effective integration of multiple documents is crucial. The research also highlighted R2U’s generalizability across diverse generators and model sizes, showing that even smaller models, which typically lack strong reasoning capabilities, benefit significantly from R2U’s effectiveness.
In conclusion, R2U offers a powerful solution to bridge the gap between retrieval relevance and generative utility in RAG systems. By directly observing and distilling the reasoning processes of large language models, it ensures that retrieved documents are not only relevant but truly useful for generating accurate and robust answers, thereby enhancing the overall performance of RAG systems across various complex question-answering tasks.
