TLDR: ScaleDoc is a novel system designed to efficiently execute LLM-based semantic predicates over large document collections. It addresses the high computational cost of LLMs by decoupling the process into an offline phase, where document embeddings are pre-computed, and an optimized online filtering phase. In the online phase, a lightweight, query-aware proxy model is trained using contrastive learning to filter most documents, forwarding only ambiguous cases to a powerful oracle LLM. This adaptive cascade mechanism, combined with robust calibration, achieves over a 2x end-to-end speedup and reduces expensive LLM invocations by up to 85%, making large-scale semantic analysis practical and cost-effective.
In the rapidly evolving landscape of data analysis, the ability to understand and query vast collections of unstructured documents based on their semantic meaning has become paramount. Traditional methods, which rely on simple keyword matching or value-based predicates, fall short when faced with complex, nuanced queries. While Large Language Models (LLMs) offer powerful zero-shot capabilities for semantic understanding, their high inference costs and computational demands make them impractical for large-scale document analysis, especially for ad-hoc queries across millions of documents.
Addressing this significant challenge, a new system called ScaleDoc has been introduced. ScaleDoc aims to make LLM-based semantic analysis practical and efficient by intelligently decoupling the predicate execution process. This innovative approach breaks down the task into two distinct phases: an offline representation phase and an optimized online filtering phase.
The Offline Representation Phase: Building a Semantic Foundation
The journey with ScaleDoc begins with a one-time, compute-intensive offline phase. During this stage, a smaller-scale LLM (for example, one with 7 billion parameters) is used to generate a rich semantic embedding for each document in the collection. These embeddings are essentially dense vector representations that capture the deep meaning and context of the documents. Once generated, these embeddings are stored, ready for subsequent online use. This pre-computation is a crucial step, as it front-loads the most expensive LLM computations. By creating these high-quality, semantically rich document representations once, ScaleDoc eliminates the need to repeatedly run the LLM for every new query, allowing for efficient reuse across countless ad-hoc analytical tasks.
The Online Filtering Phase: Smart and Efficient Query Processing
When an ad-hoc query arrives, ScaleDoc transitions to its online phase, which employs a sophisticated proxy-cascade architecture. Instead of sending every document to a powerful, expensive LLM, ScaleDoc first trains a lightweight, query-specific proxy model. This model is trained using a small sample (e.g., 5%) of the documents, for which ground-truth labels are obtained from a powerful oracle LLM like GPT-4o. The proxy model then rapidly evaluates the pre-computed semantic embeddings of the documents, assigning each a decision score that indicates its likelihood of satisfying the semantic predicate of the query.
The core intelligence of the online phase lies in its cascade filter. Based on the proxy’s decision scores, documents are categorized into high-confidence positive, high-confidence negative, or ambiguous. The high-confidence documents are filtered out, with their classifications accepted directly from the proxy. Only the ambiguous cases, those with scores falling within a specific range, are forwarded to the powerful oracle LLM for a final, accurate judgment. This hierarchical approach strategically minimizes invocations of the expensive oracle LLM, ensuring scalable and cost-effective query execution while maintaining high accuracy.
Innovations for Efficiency and Accuracy
ScaleDoc’s significant efficiency gains are driven by two core innovations. The first is a contrastive-learning-based framework designed to train the lightweight proxy model. Traditional binary classifiers often struggle to capture query-specific semantics and produce unreliable scores. ScaleDoc’s framework ensures that the proxy model generates decision scores with desirable properties: smoothness, semantic monotonicity (higher scores for more semantically satisfying documents), and bipolarity (clear separation between positive and negative document clusters). This is achieved through a two-phase training strategy that uses specific loss functions to first establish semantic relationships and then enforce a strong separation between classes.
The second innovation is an adaptive cascade mechanism. In ad-hoc settings, determining the optimal filtering thresholds to meet a user-specified accuracy target while minimizing LLM calls is challenging due to unknown data distributions. ScaleDoc addresses this with a robust calibration workflow. This workflow uses stratified sampling to ensure representative data from all score regions, followed by distribution reconstruction techniques (jittering, linear interpolation, and smoothing) to accurately model the global score distributions from a small sample. An optimized algorithm then efficiently selects the ideal lower and upper bounds for filtering, guaranteeing accuracy while maximizing data reduction.
Also Read:
- MMORE: Bridging the Gap Between Diverse Data and Large Language Models
- Enhancing LLM Accuracy with InfoGain-RAG: A New Approach to Document Filtering and Reranking
Impressive Performance Gains
Evaluations across three diverse datasets—BigPatent, PubMed, and GovReport—demonstrate ScaleDoc’s superior performance. The system achieves an average end-to-end speedup of over 2 times and reduces expensive LLM invocations by up to 85%. This translates to significant cost savings and makes large-scale semantic analysis practical and efficient. The breakdown of latency shows that ScaleDoc’s lightweight proxy model, both in training and inference, contributes significantly to its efficiency, unlike LLM cascade methods that still incur high computational costs from smaller LLM proxies.
ScaleDoc represents a significant advancement in leveraging LLMs for large-scale document analysis. By intelligently orchestrating offline representation and online filtering with novel training and calibration mechanisms, it overcomes the computational barriers of LLMs, opening new possibilities for semantic understanding across vast document collections. For more detailed information, you can refer to the original research paper.
