TLDR: This research introduces Parallel-Distill-Refine (PDR), an inference framework that allows Large Language Models (LLMs) to achieve higher accuracy with lower latency and context length compared to traditional long chains of thought. PDR works by generating diverse drafts in parallel, distilling them into a compact summary, and then refining the output iteratively. The paper also proposes an operator-consistent Reinforcement Learning (RL) training method to align model training with this iterative inference process, leading to further performance gains on complex math tasks.
Large Language Models (LLMs) have shown remarkable capabilities in complex reasoning tasks, often by generating extensive “chains of thought” (CoT). While these long reasoning traces can lead to higher accuracy, they come with significant drawbacks: increased context length, higher token and compute costs, and longer answer latency. A new research paper, “Rethinking Thinking Tokens: LLMs as Improvement Operators”, explores a novel approach to overcome these limitations by viewing LLMs as “improvement operators” on their own thoughts.
The core idea is to enable models to leverage their metacognition – their ability to think about their own thinking – to achieve better accuracy without the inflated costs of very long reasoning sequences. Instead of a single, lengthy thought process, the paper proposes iterative strategies that allow LLMs to refine their answers in a controlled, efficient manner.
The Challenge with Long Chains of Thought
Traditional long CoT methods, where LLMs produce detailed step-by-step reasoning, often entangle reasoning depth with the sheer length of the generated sequence. This can lead to “long-context failure modes,” where the model struggles to maintain coherence or utilize information effectively over very long inputs. Moreover, the computational expense and time taken for these long traces are substantial, making them less practical for real-world applications requiring quick responses.
Introducing Iterative Improvement Operators
The researchers introduce an inference family called Parallel-Distill-Refine (PDR), which offers a new way for LLMs to approach problem-solving. PDR breaks down the reasoning process into manageable, iterative rounds, each with three key steps:
- Parallel Generation: The model generates multiple diverse draft solutions or reasoning paths simultaneously. This allows for broad exploration of solution strategies.
- Distillation: These diverse drafts are then condensed into a compact, bounded textual workspace. This workspace acts as a summary, capturing agreements, contradictions, intermediate results, and open subgoals from the parallel drafts. Crucially, it keeps the memory bounded, preventing context length from spiraling out of control.
- Refinement: Conditioned on this compact workspace, the model refines its output, producing an improved answer that then seeds the next round of the process.
This approach ensures that context length, and therefore compute cost, is controllable and no longer directly tied to the total number of generated tokens. The paper also examines a subcase of PDR called Sequential Refinement (SR), where a single candidate answer is iteratively improved over several rounds.
Key Advantages and Findings
The experiments, conducted on challenging math tasks like AIME 2024 and AIME 2025, demonstrated significant advantages for PDR and SR:
- Improved Accuracy and Latency: PDR instantiations of current models (like gpt-o3-mini and gemini-2.5-flash) achieved better accuracy than long CoT while incurring lower latency. For instance, on AIME 2024, PDR showed an absolute improvement of +11% over long CoT, and +9% on AIME 2025.
- Efficient Context Management: By using a round-wise, non-persistent summary, PDR avoids the long-context failure modes and scaling costs associated with appending all prior attempts to the context.
- Effective Distillation Strategies: The study compared different ways to construct the compact summary, finding that “global summary” (aggregating all candidates into a single summary) and “per-sample top-k” (each downstream branch selecting its own top-k candidates) generally performed best.
- Impact of Verification: The research highlighted the importance of the model’s self-verification abilities. Injecting incorrect candidates into the summary significantly degraded performance, especially for models with weaker intrinsic self-verification.
Operator-Consistent Training
Beyond just orchestrating inference, the paper also addresses the “train-test mismatch.” Most Reinforcement Learning (RL) training for reasoning LLMs optimizes a single, long chain-of-thought trajectory. However, if inference uses multiple short passes with a compact workspace (as in PDR), this creates a discrepancy. To resolve this, the researchers developed an operator-consistent RL training strategy. This method mixes standard long-trace optimization with “operator rollouts” that explicitly train the model on the generate-distill-refine interface under short contexts. This approach further boosted performance, yielding approximately +5% gains on AIME 2024 and AIME 2025, demonstrating that models can learn the meta-skills necessary for effective iteration.
Also Read:
- DIVER: A New Approach to Enhance LLM Reasoning Through Diverse Exploration
- Chain-in-Tree: Streamlining LLM Reasoning with Adaptive Branching
Conclusion
This research marks a significant step in exploring a broader design space for LLM reasoning beyond traditional long chains of thought. By introducing Sequential Refinement (SR) and especially Parallel-Distill-Refine (PDR), the authors show that iterative, compact-memory approaches can outperform long-trace baselines in terms of accuracy while maintaining or even reducing latency. The findings suggest that by focusing on diversity, verification, and refinement within a bounded context, and by aligning training with these iterative inference methods, LLMs can achieve more intelligent and efficient problem-solving capabilities.
