TLDR: This research introduces a unified scaling law for merging large language models (LLMs), demonstrating how cross-entropy loss decreases predictably with model size (N) and the number of merged experts (k). The law reveals that larger models are easier to merge, most performance gains occur with a few experts, and merging methods converge in effectiveness at scale. This transforms model merging from a trial-and-error process into a computationally efficient and planable alternative to multitask training, enabling predictive planning for AI development.
Large Language Models (LLMs) have become incredibly powerful, often specialized for different tasks through a process called fine-tuning. This creates multiple ‘expert’ models, each proficient in a specific domain. A common practice is ‘model merging,’ where these experts are combined in their weight space to create a single, more versatile model without the need for extensive retraining. This approach is attractive because it can approximate the performance of training a single model on multiple tasks, but at a significantly lower computational cost.
Despite its widespread use, model merging has largely remained an empirical art, lacking the quantitative rules that guide other areas of AI development, such as pre-training. Practitioners often resort to trial-and-error, experimenting with different subsets of experts, merging orders, and normalization techniques, which can be computationally expensive and unpredictable.
Unveiling a Unified Scaling Law
A groundbreaking new research paper, titled “MODEL MERGING SCALING LAWS IN LARGE LANGUAGE MODELS,” by Yuanyi Wang, Yanggan Gu, Yiming Zhang, Qi Zhou, Zhaoyi Yan, Congkai Xie, Xinyao Wang, Jianbo Yuan, and Hongxia Yang, addresses this gap by introducing a compact, predictive scaling law for language model merging. This law quantitatively links the model’s base size (N) with the number of merged experts (k) to predict the cross-entropy loss, a common measure of a model’s performance.
The core of this new law is expressed as: Expected Loss = L∞(N) + A(N)/(k+b). In simpler terms, this means that the model’s performance improves in two main ways:
- Size-Dependent Floor (L∞(N)): Larger base models inherently achieve a lower minimum possible loss. This ‘floor’ of performance decreases as the model’s capacity (N) increases.
- Merging Tail (A(N)/(k+b)): As more experts (k) are added, the performance improves, but with clear diminishing returns. Most of the significant gains happen early on, and then the improvements taper off, following a roughly 1/k pattern.
This law was rigorously validated through extensive experiments involving 10,506 merged models, with base sizes ranging from 0.5 billion to 72 billion parameters, across nine diverse domains, and using four popular merging methods (Average, Task Arithmetic (TA), TIES, and DARE). The law consistently fit the measured performance curves with high accuracy (R² > 0.98).
Also Read:
- Unlocking New Abilities: How Reinforcement Learning Helps Language Models Compose Skills
- UniAPL: Unifying Language Model Training for Enhanced Instruction Following
Key Discoveries and Practical Implications
The study revealed several robust regularities:
- Larger Models, Easier Merging: Bigger base models are not only better performers at any given number of experts, but they also require fewer experts to reach their optimal performance floor.
- Early Gains are Key: The majority of performance improvements from merging experts occur very early in the process, typically after merging just 5 to 6 experts. Beyond this ‘elbow’ point, additional experts yield progressively smaller gains.
- Domain Mixing Helps: Merging experts from diverse, heterogeneous domains still adheres to the same scaling law, demonstrating that mixing domains can aid in pooled generalization.
- Method Differences Shrink at Scale: While different merging methods might show slight advantages early on, these differences become negligible as the number of experts and model size increase. All methods converge towards similar performance levels.
- Order Sensitivity Fades: The order in which experts are merged quickly becomes inconsequential as more experts are included, especially after around 6 experts.
- Architecture Agnostic: The scaling law holds true across different model architectures, as demonstrated by its consistent application to LLaMA backbones.
This research transforms model merging from a heuristic practice into a computationally efficient and planable alternative to multitask fine-tuning. The ability to predict performance allows for ‘predictive planning’ – estimating the number of experts needed to achieve a target loss, deciding when to stop adding experts, and making informed trade-offs between scaling the base model and adding more experts under a fixed budget. This suggests a powerful scaling principle for distributed generative AI: predictable gains can be achieved by composing specialists, offering a complementary path toward AGI-level systems.
For more in-depth information, you can read the full research paper here.
