TLDR: The InfinityInstruct-Subject research introduces a novel framework for creating high-quality instruction datasets that expand both the coverage and complexity of instructions for large language models (LLMs). By integrating hierarchical tagging, informative seed selection, evolutionary data synthesis, and model deficiency diagnosis, the framework generates a 1.5 million instruction dataset. This dataset significantly improves LLM performance on complex tasks, outperforming existing datasets and official instruction-tuned models. The study also reveals a scale-free topology in instruction tag co-occurrence, offering new insights into LLM scaling laws and knowledge structure.
Instruction tuning is a fundamental technique for unlocking the full potential of large language models (LLMs) and improving their ability to handle complex tasks. However, despite the existence of instruction datasets with tens of millions of samples, models fine-tuned on them often struggle with intricate instructions and tasks in less common domains. This challenge primarily stems from limitations in both the “coverage” (the variety of task types and knowledge areas) and “depth” (the complexity of instructions) within existing datasets.
To address these limitations, researchers have introduced a systematic framework for constructing instruction data, which integrates several key components. This framework includes a hierarchical labeling system, an algorithm for selecting informative seed data, an evolutionary process for synthesizing new data, and a mechanism for diagnosing model deficiencies to generate targeted data. These components work together in an iterative, closed-loop system designed to continuously enhance both the coverage and depth of instruction data.
Based on this innovative framework, a new high-quality dataset called InfinityInstruct-Subject (InfInstruct-Sub) has been developed. This dataset contains approximately 1.5 million instructions. Experiments conducted on various foundation models and benchmark tasks have demonstrated its effectiveness in significantly improving instruction-following capabilities. The models fine-tuned with InfinityInstruct-Subject consistently outperform their officially instruction-tuned counterparts.
Further analysis of the InfinityInstruct-Subject dataset reveals that it offers enlarged coverage and depth compared to other comparable synthesized instruction datasets. This is crucial because simply increasing the quantity of data does not necessarily lead to performance improvements, especially for more challenging instructions. The research highlights the necessity of enhancing the depth of the instruction set to achieve better performance.
The construction process of InfinityInstruct-Subject begins with collecting high-quality seed instructions from a large pool of existing datasets. An automatic labeling system, powered by LLMs, then analyzes the distribution of this data. Based on insights into coverage and depth, a set of high-information seed instructions is selected. An evolutionary algorithm is then applied to generate over a million new instruction samples, evolving them towards greater complexity and difficulty. Additionally, a model deficiency diagnosis system identifies gaps in model capabilities and guides the targeted synthesis of new data to efficiently address these weaknesses. A strict semantic similarity-based data leakage prevention framework is also implemented to ensure the reliability of model evaluation.
The hierarchical multilingual tagging system is a core component, assigning both Chinese and English tags at domain-level and fine-grained levels to each instruction. This helps in understanding the content and ability distribution of existing instruction content. Informative seed instruction selection focuses on instructions that are underrepresented, difficult, require multiple skills, or are poorly handled by base models, ensuring that the synthesized data expands into new and challenging areas.
An interesting finding from the data analysis is a scaling law observed in the distribution of fine-grained tag connectivity. The frequency of tags co-occurring with others follows a negative log-log linear relationship, suggesting a scale-free topology in the underlying knowledge structure of instruction data, similar to complex networks like the Internet. This implies the existence of “core” knowledge or skills that frequently co-occur with a wide range of other skills, offering new insights into how data distribution relates to model performance scaling laws.
Also Read:
- TACOS: A New Method for Smarter Data Selection in LLM Fine-Tuning
- Unstructured Reasoning Outperforms Structured Approaches in LLMs for Complex Problem Solving
This work lays a theoretical and practical foundation for the efficient and continuous evolution of instruction datasets, shifting the focus from mere data quantity expansion to qualitative improvement. For more technical details, you can refer to the full research paper here.
