TLDR: A new research paper introduces THINK LOGIT and THINK LOGIT-DPO, decoding-time techniques that enable large language models (LLMs) to perform complex, long chain-of-thought reasoning without extensive training. By using ‘logit arithmetic’ to blend the outputs of a large target model with a smaller, reasoning-trained ‘guider’ model, THINK LOGIT significantly boosts performance. THINK LOGIT-DPO further refines this by training the guider with Direct Preference Optimization (DPO) using preference pairs from both models. The methods achieved substantial improvements on mathematical reasoning benchmarks, demonstrating a computationally efficient way to enhance LLM reasoning capabilities.
Large language models (LLMs) have shown incredible potential for complex reasoning, often by generating long, detailed thought processes known as chain-of-thought (CoT). These advanced reasoning abilities typically require extensive and costly training, such as reinforcement learning or supervised distillation. However, a new research paper explores whether these capabilities can be unlocked in large models without such heavy training, by leveraging smaller, more efficient models.
The paper, titled “Logit Arithmetic Elicits Long Reasoning Capabilities Without Training,” introduces two innovative decoding-time approaches: THINK LOGIT and THINK LOGIT-DPO. These methods aim to elicit long reasoning from large, pre-trained LLMs with minimal or no additional training, offering a computationally efficient alternative.
THINK LOGIT: Guiding Large Models with Small Ones
At its core, THINK LOGIT is a technique that allows a small, specialized reasoning model to guide a much larger target model during the text generation process. It achieves this through a concept called “logit arithmetic.” Imagine the large model and the small guider model each producing a set of probabilities for the next word in a sequence. Logit arithmetic essentially combines these probabilities. Specifically, it takes the difference in logits (a measure of a model’s confidence in predicting the next token) between a small reasoning-trained model and a small base model, and adds this ‘reasoning delta’ to the logits of the large target model. This subtly nudges the large model’s generation towards longer, more complex reasoning paths, mirroring the capabilities of the smaller, trained guider, all without altering the large model’s internal structure or weights.
THINK LOGIT-DPO: Refining the Guidance
While THINK LOGIT is effective, the researchers found that the output styles of the guider and target models might not always perfectly align. To address this, they developed THINK LOGIT-DPO. This enhanced approach involves an additional step: training the small guider model using Direct Preference Optimization (DPO). DPO is a method that optimizes a model based on preferences, in this case, pairs of correct and incorrect reasoning examples. These examples are carefully selected from both the large target model and the small guider model, capturing their complementary strengths. By training the guider with DPO, its guidance becomes more aligned with the target model’s behavior, leading to even better performance.
Impressive Results on Mathematical Reasoning
The effectiveness of THINK LOGIT and THINK LOGIT-DPO was evaluated on several mathematical reasoning datasets, including AIME2024, AIME2025, AMC23, and a challenging subset of MATH500. The target model used was Qwen2.5-32B, a large language model, guided by a significantly smaller R1-Distill-Qwen-1.5B model (which is 21 times smaller). The results were notable:
- THINK LOGIT achieved an average relative improvement in pass@1 (a metric for correct answers on the first attempt) by 26% over the Qwen2.5-32B baseline.
- THINK LOGIT-DPO further boosted this, reaching a 29% relative improvement.
- The research also demonstrated that THINK LOGIT could transfer reasoning skills acquired through reinforcement learning, improving pass@1 by 13% relative compared to the base model, without any RL training on the large model itself.
These findings suggest that these lightweight, inference-time techniques can recover a substantial portion of the gains typically achieved by full-parameter fine-tuning, but with only a fraction of the data, computational cost, and parameter updates. The paper highlights a practical and efficient way to deploy LLMs with long chain-of-thought reasoning capabilities.
Also Read:
- Dynamic Tree Reasoning with Reinforcement Learning for Adaptive LLM Problem Solving
- Boosting LLM Reasoning: A New Approach with Question Augmentation
Looking Ahead
The work by Yunxiang Zhang, Muhammad Khalifa, Lechen Zhang, Xin Liu, Ayoung Lee, Xinliang Frederick Zhang, Farima Fatahi Bayat, and Lu Wang from the University of Michigan, Ann Arbor, presents a compelling case for eliciting complex reasoning in large models without the traditional training burden. This approach opens doors for more accessible and efficient development of advanced AI systems. For more details, you can read the full research paper here.
