TLDR: CompactPrompt is a novel pipeline designed to reduce the operational costs of Large Language Models (LLMs) by compressing both prompts and attached data. It achieves this by pruning low-information tokens from prompts using self-information scoring and dependency-based phrase grouping, and by applying n-gram abbreviation to text and uniform quantization to numerical data. This method can cut token usage by up to 60% on benchmarks like TAT-QA and FinQA, while maintaining or even improving output accuracy for models such as Claude-3.5-Sonnet and GPT-4.1-Mini. The pipeline helps mitigate issues like the ‘lost-in-the-middle’ effect and offers a training-free, end-to-end solution for more efficient LLM workflows.
Large Language Models (LLMs) have become incredibly powerful tools, capable of complex reasoning and generating human-like text. However, using them in real-world applications, especially in agentic workflows that involve long prompts and rich data streams, can be quite expensive and computationally intensive. This is where a new solution called CompactPrompt comes into play, offering a unified approach to make LLM operations more efficient.
CompactPrompt is an end-to-end pipeline designed to tackle the challenge of high run-time costs by merging two key strategies: hard prompt compression and lightweight file-level data compression. The goal is to reduce the total number of tokens used by LLMs, thereby lowering inference costs, while still maintaining the quality and accuracy of the output.
How CompactPrompt Works
The pipeline operates in two main phases:
1. Hard Prompt Compression: This part focuses on the prompts themselves. CompactPrompt intelligently prunes tokens that carry low information. It does this by scoring tokens based on their self-information and grouping phrases based on dependencies. This ensures that only the most crucial parts of the prompt are retained, making it shorter without losing its core meaning.
2. Data Compression: For any attached documents or data streams, CompactPrompt employs two distinct methods:
- N-gram Abbreviation: For recurrent textual patterns, such as common phrases or domain-specific terminology, CompactPrompt uses n-gram abbreviation. This technique identifies high-frequency multi-word expressions and replaces them with unique, shorter, human-readable tokens. This process is reversible, meaning the original text can be fully reconstructed if needed.
- Numerical Quantization: For numerical data, especially in structured columns, uniform quantization is applied. This maps floating-point numbers to fixed-bit integer codes, significantly reducing token usage for numerical data while preserving analytical accuracy within specified error tolerances. K-means based quantization is also an alternative.
This integrated approach allows CompactPrompt to yield compact yet semantically faithful representations of both prompts and attached data.
Addressing LLM Limitations
The research highlights that beyond a certain length, adding more tokens can actually degrade an LLM’s reasoning performance. This is partly due to the “lost-in-the-middle” effect, where transformer attention mechanisms tend to over-emphasize tokens at the beginning and end of a prompt, neglecting information in the middle. By compressing prompts, CompactPrompt helps LLMs operate within their effective working window, improving their ability to integrate all relevant context during multi-step reasoning tasks.
Performance and Results
CompactPrompt has been rigorously evaluated on benchmark datasets like TAT-QA and FinQA, which involve complex question answering over financial tables and text. The results are impressive:
- It reduces total token usage and inference cost by up to 60%.
- It preserves output quality, showing less than a 5% accuracy drop for models like Claude-3.5-Sonnet and GPT-4.1-Mini. In many cases, it even led to measurable improvements in accuracy.
- For instance, on FinQA, Claude 3.5 Sonnet saw an improvement from 66% to 76% accuracy, and GPT-4-Omni also achieved notable gains.
The study also found that careful tuning of abbreviation hyperparameters (specifically, using bi-grams with the top 3 most frequent patterns) is crucial for balancing efficiency and fidelity. Additionally, selecting representative examples for few-shot prompting further amplifies the benefits, stabilizing performance and enabling consistent gains across various LLMs.
Also Read:
- MemCom: Enhancing In-Context Learning Efficiency Through Layer-Wise Prompt Compression
- Capsule Prompt Tuning: Efficiently Guiding Large Language Models
Availability and Future Directions
CompactPrompt is available as both a Graphical User Interface (GUI) tool and a library implementation (UnifiedCompactPromptPipeline or UCPP), allowing for seamless integration into enterprise-scale AI infrastructure, particularly in financial technology stacks. The library version enables programmatic access, making it suitable for automated trading systems, risk management platforms, and regulatory compliance pipelines.
Looking ahead, the researchers envision several promising extensions, including adaptive compression that adjusts based on task complexity, privacy-aware compression to filter sensitive information, and expanding compression techniques to multimodal contexts like charts and PDF documents. The paper also suggests that compression could reduce reliance on Retrieval-Augmented Generation (RAG) by allowing larger proprietary documents to be directly incorporated into prompts.
CompactPrompt represents a significant step towards creating leaner and more cost-effective generative AI pipelines, especially for industries like finance where operational efficiency and consistent output quality are paramount. You can learn more about this research in the full paper: CompactPrompt: A Unified Pipeline for Prompt and Data Compression in LLM Workflows.
