TLDR: This research introduces InPars+, an extension to existing synthetic data generation pipelines for Neural Information Retrieval (NIR) systems. It tackles the high cost and inefficiency of training data creation by proposing two main improvements: fine-tuning query generator LLMs using Contrastive Preference Optimization (CPO) to produce higher-quality queries, and employing dynamic, Chain-of-Thought (CoT) optimized prompts via the DSPy framework. These extensions significantly reduce the need for aggressive data filtering, improving computational efficiency and retrieval performance, making synthetic data generation more effective and less wasteful.
Training advanced search systems, known as Neural Information Retrieval (NIR) models, typically requires vast amounts of human-annotated data. This data, which links documents with relevant queries, is incredibly expensive and time-consuming to create. For instance, a human annotator might spend at least a minute assessing just one query-document pair. This high cost has led researchers to explore using large language models (LLMs) to generate synthetic, or artificial, queries.
Previous efforts in this area include methods like InPars, InPars-V2, and Promptagator. These approaches, often called QGen pipelines, involve an LLM generating queries for documents, which are then used to fine-tune a document ranking model. While promising, these methods have faced challenges. Reproducing their results can be difficult because they sometimes rely on models that aren’t publicly available or require specific hardware. A significant drawback is that a large portion of the generated synthetic data, sometimes as much as 90%, is discarded because it’s not considered high-quality enough for training. This filtering process is computationally intensive and wasteful.
A new research paper, titled “InPars+: Supercharging Synthetic Data Generation for Information Retrieval Systems,” addresses these limitations by building upon the InPars Toolkit, an open-source framework for synthetic data generation. The authors, Matey Krastev, Miklos Hamar, Danilo Toapanta, Jesse Brouwers, and Yibin Lei from the University of Amsterdam, introduce two key extensions to make synthetic data generation more efficient and effective.
The first extension involves fine-tuning the query generator LLM using a technique called Contrastive Preference Optimization (CPO). The goal here is to train the LLM to produce better quality queries from the start, reducing the amount of “noisy” or irrelevant data. This means less need for aggressive filtering later on, saving significant computational resources. They used a smaller Llama 3.1 8B model as a ‘student’ and a larger Llama 3.1 70B model as a ‘teacher’ to guide the learning process, aiming for the student model to generate queries that are more relevant and useful.
The second extension focuses on improving how LLMs are prompted to generate queries. Instead of using static, pre-defined prompt templates, InPars+ employs dynamic, Chain-of-Thought (CoT) optimized prompts through the DSPy framework. This allows the LLM to reason more effectively about the document’s content before formulating a query, leading to more contextually appropriate and higher-quality outputs. This adaptive prompting strategy helps the LLM determine what constitutes a relevant query on its own.
The researchers also validated the reproducibility of the original InPars Toolkit, confirming its ability to provide an end-to-end pipeline and its ‘plug-and-play’ functionality for integrating different LLMs. Their experiments, particularly on the SciFact dataset, showed that their extensions generally improved retrieval performance and, crucially, reduced the reliance on heavy filtering. For example, the CoT prompting strategy, even with a smaller Llama 3.1 8B model, achieved performance comparable to original InPars-V2 results while using only a quarter of the queries, demonstrating significant efficiency gains.
While the study found that filtering from larger pools of generated queries generally improved quality, the new methods aimed to make the initial generation so good that less filtering is needed. The use of dynamic prompts with CoT consistently improved performance across various datasets, highlighting the benefits of a more intelligent prompting approach. However, the paper also notes that bigger LLMs don’t always guarantee better results, and the impact of filtering can vary depending on the dataset.
Also Read:
- Evaluating RAG Systems: When Can Synthetic Data Be Trusted?
- Refining Knowledge Graph Searches with User-Defined Preferences
In summary, InPars+ offers a significant step forward in synthetic data generation for information retrieval. By fine-tuning generator models with CPO and using dynamic, reasoning-enhanced prompts via DSPy, the approach aims to create higher-quality synthetic data more efficiently, ultimately making it easier and less costly to train powerful search systems. All code, models, and synthetic datasets from this study are publicly available for further research. You can find the full research paper here: InPars+: Supercharging Synthetic Data Generation for Information Retrieval Systems.
