TLDR: PRISM is a new method for scientific paper retrieval that moves beyond abstract-based search by using full paper content. It optimizes queries by breaking them down into aspect-specific views (motivation, method, experiment) using LLM agents and matches them against fine-grained, segmented representations of candidate papers. A new benchmark, SciFullBench, was created for evaluation. PRISM significantly outperforms existing baselines, demonstrating the effectiveness of its multi-aspect query optimization and fine-grained document representation.
Finding relevant scientific papers can be a challenging task, especially when your query isn’t just a few keywords but an entire research paper itself. Traditionally, systems have relied on abstracts to find similar documents, but abstracts often provide only a high-level summary, missing the deeper, fine-grained details within a full paper.
A new approach called PRISM (Fine-Grained Paper-to-Paper Retrieval with Multi-Aspect-Aware Query Optimization) aims to solve this by looking at scientific papers in a much more detailed way. Instead of treating a paper as a single block of text, PRISM breaks down both the paper you’re using as a query and the papers in the database into multiple, specific parts.
How PRISM Works
PRISM tackles the problem from two main angles: optimizing the query and representing the candidate papers. On the query side, it uses advanced language models, acting like specialized agents, to analyze your query paper from different perspectives. For example, one agent might focus on the research’s motivation, another on the proposed methods, and a third on the experimental setup. Each of these agents then generates a specific query based on its assigned aspect. This ensures that the system looks for relevance across various dimensions of a paper, not just a general overview.
For the vast collection of candidate papers, PRISM doesn’t just store their abstracts or even their full text as single units. Instead, it segments each paper into smaller, fixed-length chunks. This allows the system to match the aspect-specific queries to the most relevant parts of the candidate papers. Imagine searching for a paper based on a specific experimental technique; PRISM can pinpoint papers that discuss that technique in detail, even if it’s buried deep within the document.
Once individual searches are performed for each aspect-specific query against the segmented candidate papers, PRISM uses a technique called Reciprocal Rank Fusion. This method intelligently combines the results from all the different searches, giving more weight to papers that appear high in multiple aspect-specific rankings. This ensures that the final list of retrieved papers is comprehensive and reflects multifaceted relevance.
Introducing SciFullBench
To properly test PRISM’s capabilities, its creators also developed a new benchmark called SciFullBench. Existing benchmarks were mostly designed for abstract-based retrieval, which wasn’t suitable for evaluating full-paper retrieval. SciFullBench includes complete papers for both queries and retrieval targets, covering machine learning and natural language processing domains. This new dataset allows for a more realistic evaluation of systems designed to understand the full context of scientific literature.
Performance and Insights
Experiments on SciFullBench show that PRISM significantly improves retrieval performance compared to existing methods. Even when using general-purpose embedding models (which aren’t specifically trained for scientific papers), PRISM achieves substantial gains. This highlights that the way queries are structured and how candidate papers are represented can make a huge difference, even more than just using longer inputs or chunked data alone.
The research also found that using multiple specialized agents for query optimization leads to better results than a single agent trying to cover all aspects. This confirms the importance of capturing the diverse nature of scientific papers. Furthermore, segmenting candidate documents into finer-grained chunks (like 3K-token segments) yielded the best performance, suggesting that detailed representations are crucial for identifying specific signals within long papers.
Also Read:
- CUE-RAG: Boosting LLM Accuracy and Efficiency with Advanced Graph-Based Retrieval
- A New Benchmark for AI’s Role in Research Experiment Design
Looking Ahead
While PRISM marks a significant step forward in scientific paper retrieval, the authors acknowledge areas for future improvement. This includes further specializing and coordinating the language model agents, and exploring more intelligent, content-aware ways to segment documents. The goal is to continue refining systems that can move beyond surface-level understanding to truly grasp the depth and breadth of scientific literature.
You can read the full research paper here: PRISM: Fine-Grained Paper-to-Paper Retrieval with Multi-Aspect-Aware Query Optimization.
