TLDR: StorInfer is a system that speeds up large language model (LLM) responses by pre-calculating and storing common questions and their answers. When a user asks a question, StorInfer quickly checks its stored answers. If a match is found, it instantly provides the precomputed response, avoiding slow LLM inference and reducing costs, especially for devices with limited resources. This approach significantly reduces latency and compute costs while maintaining response quality.
Large Language Models (LLMs) have become incredibly powerful tools, capable of understanding and generating human-like text for a wide array of applications, from summarizing documents to powering voice assistants. However, deploying these sophisticated models, especially on devices with limited resources like smartphones or edge devices, often comes with a significant challenge: high latency. This means users might experience noticeable delays in getting responses, which can be frustrating for real-time applications.
Traditional methods to speed up LLMs, such as caching previous computations, have their limitations. They often suffer from a “cold start” problem, meaning they aren’t helpful for new or slightly different queries, and they typically only optimize parts of the inference process, not the entire response time.
Introducing StorInfer: A Smart Approach to Faster LLM Responses
To tackle these challenges, researchers have introduced StorInfer, a novel system that leverages storage to dramatically accelerate LLM inference. Imagine having a vast library of pre-answered questions. When you ask a question, instead of having to figure out the answer from scratch every time, you can quickly look it up in this library. That’s the core idea behind StorInfer.
StorInfer works by pre-computing and storing anticipated query-response pairs offline. This means that before a user even asks a question, StorInfer has already generated and saved many potential questions and their high-quality answers. When a user’s query comes in and semantically matches one of these precomputed questions, StorInfer bypasses the computationally expensive process of running the LLM on a GPU and instantly returns the stored response. This significantly cuts down on latency and computational costs.
How StorInfer Works: Two Key Phases
The system operates in two main phases:
1. The Generator (Offline Phase): This is where the magic of anticipation happens. StorInfer uses a powerful LLM to generate a diverse set of user-like queries from an existing knowledge base. To ensure these generated queries are useful and not redundant, it employs two clever techniques:
- Adaptive Query Masking: This prevents the system from generating very similar queries repeatedly, ensuring a wider range of unique questions are covered.
- Adaptive Sampling: This dynamically adjusts how the LLM generates queries to promote semantic diversity, making sure the stored questions cover a broad spectrum of user intents.
Once these unique query-response pairs are created, they are embedded into high-dimensional vectors and stored in a disk-backed vector database, ready for fast, similarity-based retrieval.
2. The Runtime (Online Phase): When a user submits a new query, StorInfer gets to work immediately. It performs two actions simultaneously: it searches its precomputed database for a matching query and, at the same time, it starts the normal LLM inference process. If a sufficiently similar precomputed query is found in the database, StorInfer instantly returns the stored response and cancels the ongoing LLM inference. This prevents unnecessary computation and delivers a near-instantaneous answer. If no good match is found, the LLM inference continues as usual to generate a fresh response.
Also Read:
- SpecExit: Smarter, Faster Reasoning for Large Language Models
- Enhancing Diffusion LLM Performance with Adaptive Block Sizing
Significant Performance Gains
The evaluation of StorInfer demonstrated impressive results. By generating and storing 150,000 unique precomputed query-response pairs, which occupied about 830 MB of storage, the system achieved up to a 17.3% reduction in latency without compromising the quality of responses. The vector search component, which retrieves the precomputed answers, was found to be 8.6 times faster than a full LLM inference.
This approach is particularly effective in scenarios where user queries are somewhat predictable, such as in voice assistants with limited command sets or in retrieval-augmented generation (RAG) systems operating over a fixed set of documents. StorInfer also offers flexibility, allowing high-quality responses to be precomputed using powerful LLMs offline, while lightweight models on edge devices handle user interactions, ensuring a consistent user experience even in resource-constrained environments.
StorInfer represents a promising step forward in making LLM inference more efficient and accessible, especially for on-device and edge deployments. By strategically leveraging storage, it opens up new possibilities for faster, more cost-effective, and scalable LLM applications. You can read more about this research in the paper Accelerating LLM Inference with Precomputed Query Storage.
