TLDR: PREMIR is a novel framework that significantly enhances multimodal document retrieval, especially for unseen domains or languages. It achieves this by leveraging Multimodal Large Language Models (MLLMs) to generate diverse ‘cross-modal pre-questions’ (preQs) from documents, covering both textual and visual content. These preQs provide a richer, token-level understanding, enabling PREMIR to outperform traditional methods and achieve state-of-the-art results in challenging closed-domain and multilingual scenarios without additional training.
In today’s information-rich world, documents are rarely just plain text. They often combine text with images, tables, and charts, creating a complex tapestry of information. While advanced AI models, known as Multimodal Large Language Models (MLLMs), have made strides in understanding these complex documents, a significant challenge remains: how to effectively search and retrieve information from documents that the AI has never seen before, especially in specialized fields or different languages.
Current retrieval systems often struggle in these ‘out-of-distribution’ scenarios. They typically embed entire documents or pages as single vectors, which can fail to capture the fine-grained details and cross-modal nuances (how text and images relate to each other). This leads to poor performance when faced with new types of documents or languages.
Introducing PREMIR: A Novel Approach to Multimodal Document Retrieval
A new framework called PREMIR (Pre-question Multimodal Information Retrieval) offers an elegant solution to this problem. Developed by researchers from Yonsei University and Seoul National University, PREMIR leverages the vast knowledge of MLLMs to generate ‘cross-modal pre-questions’ (preQs) from documents before any retrieval even happens. These preQs act as a rich, token-level representation of the document’s content, allowing for a much deeper contextual understanding.
Unlike older methods that try to embed an entire document into a single vector, PREMIR breaks down documents into finer, token-level representations across different modalities. This means it doesn’t just see a document as a whole, but understands the individual pieces of information within its text and visuals, and how they connect.
How PREMIR Works
The PREMIR framework operates in several key steps:
First, it performs Multimodal Document Parsing. This involves using a layout-aware parser to extract both visual and textual components from each page of a document. For instance, it identifies figures, tables, charts, and optical character recognition (OCR) text. These components are then captioned by an MLLM, and these captions are merged with the OCR text, preserving the original layout. This creates a comprehensive textual surrogate that accurately reflects the multimodal content of the page.
Next is Cross-modal PreQ Generation. From the parsed document, PREMIR generates three complementary sets of pre-questions:
- Multimodal preQs: Generated directly from the raw page image, these questions capture the overall layout and cross-modal context.
- Visual preQs: Created from individual visual components (like figures or tables), these expose modality-specific cues.
- Textual preQs: Derived from the layout-aware textual surrogate, focusing on linguistic details.
These preQs are designed to cover not only explicitly stated information but also knowledge implicitly conveyed within the document, drawing on the MLLM’s broad understanding.
Finally, Q-Cluster Retrieval takes place. When a user submits a query, PREMIR encodes it and retrieves the top-k most similar preQs. Since a single preQ might not fully capture a user’s intent, PREMIR clusters preQs that originated from the same source passage. An LLM then evaluates these clusters to select the most relevant passages, effectively increasing the likelihood of retrieving a passage that directly answers the user’s query.
Also Read:
- Enhancing Multi-Image Question Answering in AI Models with Adaptive Visual Anchoring
- Improving Product Search in Multi-Turn Conversations with Dynamic Reranking
Outstanding Performance in Real-World Scenarios
PREMIR has demonstrated state-of-the-art performance across various challenging benchmarks. It significantly outperforms strong baselines in zero-shot closed-domain multimodal document retrieval tasks, such as ViDoSeek and REAL-MM-RAG, which involve diverse topics and industrial documents. Crucially, it achieves this without any additional multimodal retrieval training.
The framework also excels in multilingual settings, outperforming other models on Chinese (CT2C-QA) and Korean (Allganize RAG) benchmarks. This robust generalization across different languages and specialized domains highlights PREMIR’s strong applicability in real-world personal and industrial applications.
Ablation studies confirm the importance of each component: the preQ-based retrieval consistently outperforms conventional page-level retrieval, combining all three preQ types (multimodal, visual, textual) yields the best results, and the Q-Cluster mechanism provides substantial performance gains by helping prioritize relevant passages.
Furthermore, PREMIR is designed for practical use. Latency and cost analyses show that while it involves more steps, optimized settings allow it to achieve competitive online latency and be more cost-efficient than some competing models for preQ generation, especially with proprietary APIs or smaller open-source models.
In conclusion, PREMIR represents a significant leap forward in multimodal document retrieval, particularly for handling unseen and complex documents. By generating intelligent, cross-modal pre-questions, it enables AI systems to understand and retrieve information with unprecedented accuracy and robustness. For more details, you can read the full research paper here.
