TLDR: UCoT is a new framework that efficiently compresses Chain-of-Thought (CoT) reasoning in Large Language Models (LLMs). It uses a small “compressor” model to generate dense “Upfront Thought” (UT) embeddings, which a larger “executor” model then uses to produce accurate answers with significantly shorter reasoning paths. This cooperative approach reduces computational costs and latency while maintaining or improving performance, achieving a 50% token reduction and 3.08% higher accuracy on GSM8K for Qwen2.5-7B-Instruct compared to SOTA methods.
Large Language Models (LLMs) have shown remarkable reasoning abilities, often by breaking down complex problems into sequential steps, a process known as Chain-of-Thought (CoT). While powerful, these long CoT sequences come with a significant drawback: high computational costs and increased processing time due to the way LLMs generate text step-by-step.
To address this efficiency challenge, researchers have been exploring ways to compress CoT. Previous attempts often involved either complex manual prompt design or creating special datasets of compressed CoTs, which sometimes led to a loss of important reasoning details or were difficult to optimize.
A new approach, called Upfront Chain-of-Thought (UCoT), offers an efficient and automated solution for CoT compression. UCoT introduces a cooperative framework that involves two models working together: a smaller “compressor” model and a larger “executor” model. This framework aims to maintain the powerful reasoning capabilities of the larger LLM while significantly shortening the length of the generated CoT.
The UCoT process unfolds in two main stages. The first stage, “Upfront Thought Generation,” involves training the smaller compressor model. Its job is to generate “Upfront Thought” (UT) embeddings. These are dense, continuous representations that capture the essential reasoning information from a high-quality, longer CoT. This method avoids the need for manually designed prompts, making the process more automated and less prone to sacrificing key reasoning details.
In the second stage, “Upfront Thought Utilization,” the larger executor model is optimized. It learns to interpret and effectively use these upfront thought embeddings. By leveraging this condensed prior knowledge, the executor can then produce the correct answer with a much shorter reasoning path. A special reward mechanism is used during training to guide the executor to understand the UTs and reduce its own output while still arriving at the correct solution.
During inference, the compressor first takes a question and generates the upfront thought embeddings. These embeddings are then passed to the executor, which uses them along with its own knowledge to generate a concise reasoning path and the final answer. This two-stage process allows for more efficient reasoning.
Extensive experiments have demonstrated the effectiveness of UCoT. For instance, when applied to the Qwen2.5-7B-Instruct model on the GSM8K dataset, UCoT reduced the token usage by 50% while actually improving performance by 3.08% compared to state-of-the-art methods. Similar improvements were observed across other models and datasets, including those requiring extra-long CoTs for complex tasks like scientific question answering (GPQA) and code generation (HumanEval).
The research also highlighted that the “information volume” within the Upfront Thoughts matters. A higher information volume in the UTs correlates with better performance, suggesting that UCoT successfully compresses rich reasoning information into these embeddings. Furthermore, increasing the length of the UTs gradually improves the executor’s accuracy, indicating that more complete reasoning information can be provided to the larger model.
Also Read:
- Deepening Knowledge Integration: How Semantic-Condition Tuning Enhances LLMs for Knowledge Graph Completion
- Optimizing LLM Performance with Intelligent Query Routing
While UCoT shows promising results, the authors note some limitations. Currently, it has only been evaluated on a limited set of open-source models, and its applicability to closed-source models is yet to be explored. The method also applies a uniform compression ratio, lacking adaptive adjustment based on problem complexity. Future work will involve evaluating UCoT on even larger-scale LLMs to further validate its scalability. For more details, you can read the full paper here.
