TLDR: Researchers from Shanghai Jiao Tong University and Alibaba Cloud have expanded the DistilQwen model family with four new series: slow-thinking models for high accuracy, two adaptive-thinking model series that adjust reasoning strategies dynamically, and distilled reward models for reinforcement learning. These models, developed using advanced knowledge distillation techniques and CoT processing, offer a balance of high inference efficiency and strong reasoning performance, and are integrated into Alibaba Cloud PAI for industrial applications.
In the rapidly evolving world of artificial intelligence, the demand for models that are not only powerful but also efficient and fast is growing. Large Language Models (LLMs) are at the forefront of many applications, but their size can often lead to slow inference speeds, making them challenging to deploy in real-world industrial settings. This is where the concept of knowledge distillation comes into play, aiming to create smaller, more agile models without significantly sacrificing performance.
A recent research paper, titled “Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Series,” introduces a significant expansion to the DistilQwen model family. Developed by researchers Wenrui Cai, Chengyu Wang, Junbing Yan, Jun Huang, and Xiangzhong Fang from Shanghai Jiao Tong University and Alibaba Cloud Computing, this work focuses on developing specialized models to meet diverse industrial requirements.
Introducing the DistilQwen Model Series
The paper details four distinct series of distilled models, each designed for specific reasoning needs. First, there are slow-thinking models, which are built for tasks where accuracy is the top priority. These models ensure consistently high precision, even if it means taking a bit more time to process.
Second, there are two series of adaptive-thinking models. Recognizing that different tasks require different levels of reasoning, these models dynamically adjust their strategies. They can be more concise for simpler problems and more elaborate for complex ones, optimizing both efficiency and performance.
Finally, there are distilled reward models. These models are crucial for further enhancing reasoning abilities through reinforcement learning. They essentially learn to predict how “good” a reasoning step is, guiding the main reasoning models to better outcomes.
The Science Behind the Models
The development process for these DistilQwen models is quite sophisticated. It begins with a “Data Source Collector” that gathers vast amounts of Chain-of-Thought (CoT) datasets from platforms like Hugging Face and ModelScope, covering diverse domains such as mathematics, code, and science. These datasets are then processed by “LLM-Based CoT Processors.”
For slow-thinking models, a CoT processor generates and refines reasoning paths using powerful teacher LLMs like DeepSeek-R1. It includes a “CoT Difficulty Scorer” to categorize reasoning steps by complexity (easy, medium, hard), allowing student models to focus on learning from medium-level examples. A “CoT Re-writer and Verifier” further refines and validates these reasoning paths.
The adaptive-thinking models utilize an even more advanced processor. This involves generating multiple CoTs from different teacher models and varying inference temperatures to increase diversity. Crucially, “Reasoning Verbosity (RV) and Cognitive Difficulty (CD) Scorers” are employed to ensure that the generated CoTs are appropriately detailed for challenging problems and align with the student models’ learning capacities. This helps models learn to “adaptively think” based on the problem’s input.
Finally, a “Distilled Model Trainer” uses supervised fine-tuning (SFT) with curriculum learning, starting with easier examples and gradually introducing more challenging ones. For the reward models, RV and CD scores are used to train lightweight reward predictors, which can then guide reinforcement learning without needing the original, much larger teacher models.
Also Read:
- DeepCompress: Adaptive AI Reasoning for Better Performance and Efficiency
- Adaptive Effort Control: AI Models Learn to Optimize Reasoning for Cost and Accuracy
Performance and Practical Applications
The models were rigorously evaluated on benchmarks covering mathematics (AIME2024, MATH500), programming (LiveCodeBench V2), and general question answering (GPQA Diamond). The results show that both slow-thinking and adaptive-thinking DistilQwen models significantly improve reasoning abilities compared to their base Qwen counterparts. The adaptive-thinking models, in particular, demonstrated superior performance, partly due to leveraging substantially more training data and their ability to generate optimal CoTs based on problem difficulty.
The distilled reward models also proved effective, enhancing reinforcement learning algorithms and leading to better performance in mathematical reasoning tasks. This suggests a promising path for future AI development.
Beyond research, these DistilQwen models are designed for practical industrial use. They are integrated into the Alibaba Cloud PAI (Platform for Artificial Intelligence) platform, offering scalable training and inference functionalities. This integration allows businesses to easily fine-tune and deploy these specialized AI capabilities for various real-world applications, from decision-making to code generation. For more details, you can refer to the full research paper here.
This work represents a significant step towards making advanced AI reasoning more accessible and efficient for industrial practitioners, balancing high performance with practical inference speeds.
