TLDR: BudgetThinker is a novel framework that enables Large Language Models (LLMs) to perform complex reasoning while strictly adhering to a specified token budget. It achieves this by inserting special “control tokens” during the reasoning process to continuously inform the model of its remaining budget. A two-stage training process, combining supervised fine-tuning and reinforcement learning with a length-aware reward, teaches the LLM to understand and utilize these budget signals. This approach significantly improves accuracy and budget adherence compared to existing methods, making LLMs more practical for real-world applications with time and cost constraints.
Large Language Models (LLMs) have shown remarkable capabilities in complex reasoning, often by generating extensive “Chain-of-Thought” (CoT) explanations before arriving at a final answer. While effective, this approach demands significant computational resources and time, making it challenging to deploy LLMs in real-world scenarios where speed and cost are critical, such as autonomous driving or AI agents.
The challenge lies in achieving controllable reasoning – enabling an LLM to tailor its thought process to conclude within a predefined computational budget. Existing methods often fall short; simply instructing an LLM about a budget in a prompt rarely guarantees adherence, and other techniques lack the fine-grained control needed for varying budget constraints.
Introducing BudgetThinker
A new framework called BudgetThinker addresses this critical need by empowering LLMs with budget-aware reasoning. The core idea is to continuously inform the model of its remaining token budget as it generates its response. This is achieved through the periodic insertion of special “control tokens” during the inference process. These tokens act as explicit signals, indicating how much of the total budget has been consumed (e.g., after 1/8th, 2/8th of the budget is used, and so on).
A Two-Stage Training Approach
BudgetThinker employs a comprehensive two-stage training pipeline to teach LLMs how to understand and respond to these control tokens:
1. Supervised Fine-Tuning (SFT): In this initial phase, the model is familiarized with budget constraints. A specialized dataset is created where each problem is assigned a budget slightly larger than the actual answer length. This teaches the model to terminate its reasoning once the answer is complete, rather than artificially padding to fill the entire budget. Crucially, the control tokens are inserted into the target outputs during this stage, allowing the model to learn their meaning.
2. Reinforcement Learning (RL): Following SFT, a curriculum-based Reinforcement Learning phase further optimizes the model. This phase uses a sophisticated reward function that balances factual accuracy, output format, and a unique “Length Reward.” The Length Reward specifically incentivizes efficient budget use, imposing a significantly higher penalty for exceeding the budget than for falling short. The training progresses from easier (larger) budgets to harder (smaller) ones, and a final mixed-budget training phase ensures the model retains proficiency across a wide range of budget levels.
Also Read:
- Enhancing Large Language Model Reasoning Through Contrastive Learning and Reinforced Fine-Tuning
- New Method Helps Large Language Models Think Smarter, Not Longer
Performance and Impact
Experiments conducted on DeepSeek distilled Qwen 2.5 1.5B and 7B models, across challenging mathematical benchmarks like MATH-500 and AMC 2023, demonstrate BudgetThinker’s effectiveness. It significantly outperforms strong baselines, including original models and other efficient reasoning frameworks, in both maintaining accuracy and precisely adhering to specified token budgets. For instance, BudgetThinker improved accuracy by an average of 4.9% across tested budgets compared to baselines.
The research also highlights that the “Budget Ratio” strategy for inserting control tokens (based on fractions of the total budget) is superior to fixed-interval insertions, as it provides a more intuitive signal for the LLM to adapt its reasoning pace. Furthermore, the Reinforcement Learning phase is shown to be vital, as it enables the model to actively explore and discover more effective reasoning strategies under budget constraints, leading to higher accuracy than SFT alone.
BudgetThinker represents a significant advancement in making LLMs more practical for real-time and resource-constrained applications. By offering precise control over the length of an LLM’s thought process, it allows for a better trade-off between reasoning quality and computational cost. You can read the full research paper here: BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens.
