TLDR: The research paper introduces CoRT (Code-Optimized Reasoning Training), a post-training framework that teaches Large Reasoning Models (LRMs) to effectively use Code Interpreters (CIs) for complex mathematical operations. CoRT utilizes Hint-Engineering, a data synthesis strategy that injects strategic hints into reasoning paths to overcome inefficiencies like delayed code computation and code result distrust. Through supervised fine-tuning, rejection fine-tuning, and reinforcement learning, CoRT significantly improves LRM accuracy (up to 8% absolute gain) and token efficiency (up to 50% reduction) across challenging mathematical benchmarks. The framework also demonstrates strong out-of-distribution generalization, with models spontaneously discovering and using unseen tools like RDKit for chemistry problems.
Large Language Models (LLMs) have shown remarkable abilities in understanding and generating human-like text, even performing complex reasoning tasks. However, when faced with intricate mathematical operations, these models often hit a wall, demonstrating inefficiencies or even inaccuracies. The challenge lies in effectively integrating external computational tools, like Code Interpreters (CIs), with the model’s internal, probabilistic reasoning. This integration often leads to a conflict, where the model might overthink or misuse the deterministic knowledge provided by the CI.
A new research paper titled “Teaching Language Models to Reason with Tools” introduces a novel solution to this problem: CoRT (Code-Optimized Reasoning Training). This post-training framework is designed to teach Large Reasoning Models (LRMs) how to effectively and efficiently utilize Code Interpreters, bridging the gap between abstract reasoning and precise computation. You can read the full paper here.
The Core Idea: Hint-Engineering
One of CoRT’s most innovative components is “Hint-Engineering,” a data synthesis strategy that strategically injects diverse hints at optimal points within the reasoning process. The researchers observed two main inefficiencies in how models typically use CIs: delayed code computation (models try to solve complex problems with text first before resorting to code) and code result distrust (models unnecessarily verify CI outputs manually). Hint-Engineering addresses these by:
- Inserting hints like “It looks tedious, and we can use python code to simplify the reasoning” when complex manual calculations are detected, prompting the model to use Python.
- Adding hints such as “We don’t need to doubt the accuracy of python calculations” when the model shows distrust in CI outputs, encouraging it to accept the deterministic results.
This targeted intervention leads to significantly shorter and more efficient reasoning paths, as the model learns when and how to best leverage the CI.
Training and Refinement
CoRT employs a multi-stage training pipeline, including supervised fine-tuning (SFT), rejection fine-tuning (RFT), and reinforcement learning (RL). The researchers synthesized 30 high-quality, human-verified samples using Hint-Engineering to post-train models ranging from 1.5 billion to 32 billion parameters. For larger models, SFT and RFT were used, while smaller models benefited from a complete SFT-RFT-RL pipeline, which allowed for direct interaction with the CI and further refinement of tool usage.
Impressive Results Across the Board
Experimental evaluations across five challenging mathematical reasoning datasets (AIME24, AIME25, AMC23, MATH500, and OlympiadBench) demonstrated CoRT’s effectiveness:
- Accuracy Improvements: CoRT yielded absolute improvements of 4% and 8% on DeepSeek-R1-Distill-Qwen-32B and DeepSeek-R1-Distill-Qwen-1.5B, respectively.
- Efficiency Gains: The framework significantly enhanced efficiency, reducing token usage by approximately 30% for the 32B model and a remarkable 50% for the 1.5B model compared to pure natural language reasoning baselines. This means models can solve problems using fewer computational resources.
- Code Behavior Shift: Analysis showed that Hint-Engineering models shifted their code usage from predominantly verification (as seen in simpler prompting methods) to a more balanced and efficient mix of direct calculation and verification. This indicates a deeper understanding of when to use computational tools.
- Reinforcement Learning Impact: For smaller models, the RL stage brought substantial improvements, demonstrating its power in refining tool-integrated reasoning.
Generalization Beyond Math
Perhaps one of the most compelling findings was CoRT’s ability to generalize to entirely new domains. In an out-of-distribution experiment on chemistry problems from the GPQA benchmark, the Hint-Engineering-RFT-32B model not only improved accuracy and efficiency but also spontaneously discovered and correctly utilized the RDKit library (a cheminformatics tool never seen during training) in over 81% of cases. This showcases a sophisticated, emergent ability to adapt to new problem contexts and identify appropriate tools on the fly, a critical step towards more general-purpose AI reasoning systems.
Also Read:
- Boosting LLM Performance with Strategy Fusion: Introducing SMaRT
- Unpacking LLM Reasoning: Relevance and Coherence as Key Metrics
Conclusion
The CoRT framework represents a significant advancement in teaching language models to reason with external tools. By combining high-quality data curation through Hint-Engineering, targeted fine-tuning, and reinforcement learning, the researchers have demonstrated that substantial improvements in both reasoning capabilities and efficiency can be achieved with minimal high-quality training data. This work paves the way for more reliable, efficient, and adaptable AI systems capable of tackling complex problems across diverse domains.
