TLDR: This research introduces SMM-Bench, the first surrogate benchmark for optimizing model merging hyperparameters. Model merging combines multiple models into one, but optimizing its settings is computationally expensive. SMM-Bench uses predictive models (LightGBM) to estimate merged model performance, drastically reducing the cost and time needed for algorithm development and comparison in both parameter space (PS) and data flow space (DFS) merging settings.
Model merging is a powerful technique that combines the strengths of several existing models into a single, more capable one. This approach is particularly valuable for enhancing large language models (LLMs) by integrating models fine-tuned for different tasks, all without needing vast datasets or expensive computations. However, most model merging techniques come with hyperparameters – settings that significantly influence the performance of the final merged model. Optimizing these hyperparameters is crucial for maximizing the merging outcome, but it’s also incredibly computationally intensive, especially when dealing with LLMs.
The process of finding the best hyperparameters for model merging, known as model merging optimization, is a type of automated machine learning (AutoML) task. While previous work has shown the effectiveness of this optimization, its high computational cost poses a significant barrier to developing new algorithms and comparing their performance fairly. Evaluating a single merging configuration can take several minutes on powerful GPUs, meaning that an optimization run involving thousands of evaluations could easily consume many GPU days. This expense slows down innovation and makes research less accessible.
Introducing Surrogate Benchmarks for Efficiency
To address this challenge, researchers have developed surrogate benchmarks. Unlike traditional benchmarks that require actual, time-consuming evaluations, surrogate benchmarks create a predictive model. This model, a regression model, can quickly estimate the performance of a merged model based on its hyperparameter settings. This allows for rapid algorithm development and comparison at a much lower cost. While tabular benchmarks offer exact evaluations, they are impractical for the continuous and large search spaces often found in model merging. Surrogate benchmarks, on the other hand, can handle these complex spaces by providing predicted performance values.
SMM-Bench: A New Tool for Model Merging Optimization
This new research introduces SMM-Bench, the first surrogate benchmark specifically designed for model merging optimization. SMM-Bench significantly reduces the evaluation cost by collecting data pairs of merging hyperparameters and their actual performance values, then training a surrogate model to predict these values. The benchmark is divided into two main settings: SMM-Bench-PS (Parameter Space) and SMM-Bench-DFS (Data Flow Space).
SMM-Bench-PS focuses on merging in the parameter space, where the parameters of multiple source models are aggregated. This involves optimizing continuous hyperparameters, such as the weighting factors in layer-wise merging, which can result in a high-dimensional search space. For example, merging two 32-layer LLMs using layer-wise task arithmetic can lead to 64 design variables.
SMM-Bench-DFS addresses merging in the data flow space, where a merged model is constructed by stacking layers from different source models. This involves a mixed category-continuous search space, including categorical variables for layer selection and continuous variables for input scaling factors.
Both benchmarks use Japanese mathematics tasks (gsm8k-ja and MGSM) to evaluate the merged models, measuring accuracy in solving mathematical problems and providing answers in Japanese. The data for training the surrogate models was collected using various strategies, including random sampling, CMA-ES, and the tree-structured Parzen estimator (TPE). LightGBM, a highly efficient gradient boosting decision tree, was chosen as the surrogate model due to its strong performance in similar benchmarks. The surrogate models demonstrated excellent predictive capabilities, with high R² scores and Kendall’s Tau coefficients, indicating they can accurately predict the performance of merged models.
Also Read:
- Enhancing Business Process Predictions with Integrated Temporal Logic
- Achieving Balance: BALM-TSF’s Strategy for LLM-Enhanced Time Series Prediction
Simulating and Accelerating Algorithm Development
The effectiveness of SMM-Bench was demonstrated by simulating optimization trajectories of different algorithms. The results showed that the surrogate benchmarks accurately capture the behavior of algorithms on true benchmarks, even when using only randomly sampled data. This capability allows researchers to compare model merging optimization methods quickly and reproducibly. For instance, a comparison between separable CMA-ES (Sep-CMA) and differential evolution (DE) on SMM-Bench-PS, which would typically require many GPU days, was completed in minutes on a laptop. This dramatic reduction in computational cost makes it feasible to conduct extensive hyperparameter tuning and comprehensive evaluations for optimizers.
In conclusion, SMM-Bench represents a significant step forward for the AutoML community by providing a low-cost, efficient, and reproducible way to develop and compare model merging optimization algorithms. This innovation promises to accelerate research and development in integrating the abilities of multiple models into a single, powerful entity. For more details, you can refer to the full research paper: Surrogate Benchmarks for Model Merging Optimization.
