TLDR: SCoder introduces an iterative self-distillation method to enable small-scale open-source language models to synthesize high-quality code instruction data. This approach significantly reduces the reliance on expensive proprietary LLMs for fine-tuning code generation models. By employing multi-checkpoint sampling, multi-aspect scoring, and gradient-based influence estimation, SCoder effectively bootstraps small models into powerful data synthesizers, leading to state-of-the-art code generation performance at a fraction of the cost.
In the rapidly evolving field of artificial intelligence, Large Language Models (LLMs) for code generation have made significant strides, enabling machines to write code that closely matches human intent. However, a major hurdle in developing these advanced models is the high cost associated with acquiring large volumes of high-quality instruction data, often distilled from expensive proprietary LLMs like GPT-3.5 and GPT-4.
A new research paper, SCoder: Iterative Self-Distillation for Bootstrapping Small-Scale Data Synthesizers to Empower Code LLMs, introduces a groundbreaking approach to tackle this challenge. Authored by Xinyu Zhang, Changzhi Zhou, Linmei Hu, Luhao Zhang, Xiancai Chen, Haomin Fu, Yang Yang, and Mengdi Zhang, this work explores the potential of smaller, open-source LLMs (such as 7B models) to act as powerful synthesizers for generating high-quality code instruction data, thereby reducing reliance on costly proprietary solutions.
The core idea behind SCoder is an innovative iterative self-distillation method. Initially, small-scale LLMs are given a boost by training them on a limited set of superior data synthesis samples derived from proprietary LLMs. This initial step demonstrates that even a small amount of high-quality data can significantly enhance the data synthesis capabilities of these smaller models.
Building on this, the SCoder framework proposes a progressive, iterative process. In each iteration, the small LLM synthesizers generate new code instruction data. To ensure this self-generated data is both diverse and of high quality, the researchers designed two key strategies:
Multi-Checkpoint Sampling and Multi-Aspect Scoring
Instead of relying on a single snapshot of the synthesizer, multi-checkpoint sampling involves collecting diverse problem-solution pairs from multiple stages (checkpoints) of the synthesizer’s training. This broadens the range of generated data. Subsequently, a multi-aspect scoring model evaluates these candidate pairs across various quality criteria, such as problem-solution consistency, correctness, and clarity. This scoring helps in selecting the most reliable and diverse samples.
Also Read:
- Optimizing Large Language Models for Clinical Data Extraction
- Optimizing LLM Training: The Power of Instruction Coverage and Depth
Gradient-Based Influence Estimation
After initial selection, a gradient-based influence estimation method is employed. This technique identifies the most influential samples by comparing their impact on the model’s training with that of the original, superior proprietary LLM samples. Essentially, it helps pinpoint which self-distilled samples are most effective in improving the model’s performance, ensuring that only the most valuable data is used for further training.
By iteratively applying these steps, the small-scale LLMs continuously improve their ability to synthesize high-quality code instruction data. This process effectively bootstraps them into powerful synthesizers, minimizing the need for expensive proprietary LLM distillation.
The effectiveness of this method is demonstrated through the development of SCoder models, which are a family of code generation models fine-tuned from DeepSeek-Coder. These SCoder models, trained on data generated by the bootstrapped small-scale synthesizers, achieve state-of-the-art code generation capabilities. They either match or outperform other leading open-source LLMs across various benchmarks like HumanEval, MBPP, LiveCodeBench, and BigCodeBench, often with significantly less reliance on proprietary data.
The research highlights that SCoder not only achieves impressive performance but also offers substantial cost efficiency. By reducing the need for extensive proprietary LLM queries, it makes the development of high-performance code LLMs more accessible and affordable for a wider range of researchers and developers. This work represents a significant step towards democratizing access to advanced code generation capabilities.
