TLDR: A new framework called DED (Data-Efficient Distillation) allows large language models to achieve state-of-the-art reasoning capabilities in math and code with significantly less training data (around 800 examples). It focuses on selecting the right teacher model, curating a small but high-quality dataset, and generating diverse problem-solving examples, challenging the traditional belief that more data or larger models are always better for reasoning. The research highlights that factors like token entropy and latent representation stability are more crucial than data quantity or response length for effective knowledge transfer.
Large language models (LLMs) have shown impressive abilities in complex tasks like algorithmic coding and solving mathematical problems. Traditionally, improving these reasoning skills has involved expanding training data or using multi-stage training methods, often leading to high computational costs. However, a new research paper introduces a groundbreaking approach that challenges this trend, proposing a data-efficient distillation framework (DED) that significantly enhances reasoning capabilities without the need for massive datasets.
The paper, titled “Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning”, was authored by a team including Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, and Jun Wang from Zhongxing Telecom Equipment (ZTE) and China Mobile Communications Group Co Ltd.
The core idea behind the DED framework is to optimize the process of distilling reasoning skills into smaller models in a data-efficient manner. Inspired by reinforcement learning principles, the framework focuses on three key innovations:
Smart Teacher Selection
Unlike conventional methods that simply pick the highest-performing LLM as a teacher, DED recognizes that raw model strength doesn’t always equate to effective teaching ability. The framework includes a practical strategy to select an optimal teacher model through a ‘smoke test’ – quick distillation trials and evaluations on reasoning benchmarks to identify which teacher truly helps the student model learn best.
Curated Corpus Design
While scaling up distillation data can improve reasoning, it often harms the model’s performance on tasks outside its specific training domain. DED addresses this by carefully curating a smaller, more targeted dataset. This approach achieves a balanced trade-off, enhancing in-domain reasoning without degrading general capabilities. The framework filters out low-quality or too-easy examples, focusing on challenging and relevant data.
Also Read:
- Teaching LLMs to Be Concise: A New Approach to Efficient Reasoning
- Boosting LLM Reasoning: A New Approach to Overcome Learning Plateaus
Diverse Reasoning Trajectories
To encourage robust reasoning skills in the student model, DED emphasizes generating diverse problem-solving paths for each question. This is inspired by the diverse roll-out strategies seen in reinforcement learning, ensuring the student model learns multiple ways to approach and solve problems, rather than just memorizing a single solution path.
The researchers validated their DED framework through extensive evaluations on various benchmarks, including mathematical reasoning tasks (AIME 2024/2025, MATH-500) and code generation (LiveCodeBench). Remarkably, the DED-trained models achieved state-of-the-art results using only about 800 carefully curated examples, demonstrating that it’s possible to bypass the need for extensive data scaling.
For instance, in mathematical reasoning, DED achieved notable accuracy rates of 81.87% and 77.29% on AIME 2024 and AIME 2025 respectively, outperforming models trained with larger datasets and even some of the teacher models themselves. Similarly, for code generation, DED showed significant improvements, especially on medium and hard difficulty levels of the LiveCodeBench, again with a highly compressed dataset.
Beyond just performance, the systematic analysis conducted by the researchers revealed deeper insights. They found that superficial factors like the length of the training corpus or the generated responses do not significantly influence distillation performance. Instead, more critical factors are the ‘token entropy’ of the corpus (how predictable and structured the token distribution is) and the ‘PCA shift’ in latent representations (indicating stability in the model’s internal understanding). Lower token entropy and smaller PCA shifts were associated with better learning and generalization, particularly for out-of-domain tasks.
In conclusion, the DED framework offers a practical and efficient pathway to developing advanced reasoning capabilities in LLMs. By focusing on smart teacher selection, meticulous data curation, and diverse reasoning examples, it challenges the conventional wisdom of ‘more is better’ in AI training, proving that quality and strategic design can indeed surpass sheer quantity, while also preserving the model’s general capabilities across diverse tasks.
