TLDR: K-Merge is a novel method for efficiently managing and merging Low-Rank Adapters (LoRAs) on mobile devices with limited storage. It addresses the challenge of incrementally arriving LoRAs by intelligently selecting and merging them with existing adapters, or allocating new slots, while preserving performance on previously supported tasks. K-Merge++ enhances this by using a similarity threshold for more robust merging decisions, optimizing storage utilization and adaptability. The approach is data-free, computationally efficient, and significantly reduces storage requirements compared to storing all individual LoRAs.
Large Language Models (LLMs) are becoming increasingly powerful, capable of handling a vast array of tasks from translation to summarization. However, deploying these massive models directly onto mobile and edge devices presents a significant challenge due to their substantial size and the limited resources available on such hardware. To overcome this, a technique called Low-Rank Adapters (LoRAs) has become popular. LoRAs are small, lightweight modules that can be added to a base LLM, allowing it to adapt to specific tasks without needing to retrain the entire large model. This means a single base LLM can support many different functions by simply loading the relevant small LoRA.
The real-world use of LoRAs on devices introduces a new hurdle: users often request support for new tasks incrementally. Imagine your phone needing a new LoRA for a novel problem type or language every now and then. Storing a separate LoRA for every single task would quickly exhaust the limited storage capacity of a mobile device. This is where the concept of ‘online continual merging’ comes into play – the need to integrate new LoRAs as they arrive, while still maintaining the performance on all previously supported tasks, all within a strict storage budget.
A new research paper, K-Merge: Online Continual Merging of Adapters for On-device Large Language Models, introduces an innovative solution to this problem. The authors, Donald Shenaj, Ondrej Bohdal, Taha Ceritli, Mete Ozay, Pietro Zanuttigh, and Umberto Michieli, propose a data-free and computationally efficient strategy for selecting and merging LoRAs on-device. Their approach assumes the device can only store a limited number of adapters, say ‘K’ adapters.
How K-Merge Works
The core of the K-Merge method involves two main components: similarity-based clustering and history-aware merging. When a new single-task LoRA arrives, the system first identifies which of the currently stored adapters is most similar to the new one. This similarity is calculated efficiently without needing any training data, by comparing the internal structures of the adapters.
Once the most similar stored adapter is found, the system makes a decision: either allocate a new storage slot for the incoming LoRA (if the device hasn’t hit its storage limit ‘K’ yet) or merge the new LoRA with the most similar existing one. The merging process is also designed to be lightweight and data-free, using a running average formulation that leverages information from previous merges. This ensures that the merged adapter retains the capabilities of both the old and new tasks, and importantly, the order in which LoRAs arrive doesn’t affect the final merged adapter’s contribution.
Introducing K-Merge++ for Enhanced Robustness
To further refine the process, the researchers developed K-Merge++. This variant introduces a ‘similarity threshold’. Instead of always merging with the closest adapter once the storage budget is full, K-Merge++ only merges if the incoming LoRA is sufficiently similar to an existing one. If it’s not similar enough, and there’s still storage available, it will be stored separately. This intelligent decision-making helps preserve storage slots for more diverse future LoRAs, making the system more adaptable and robust, especially against unpredictable task arrival orders.
Also Read:
- Unlocking Existing LoRAs: HiLoRA’s Training-Free Adaptation for LLMs
- Enhancing Sparse Large Language Models with Learnable Channel Reordering
Real-World Impact and Efficiency
The K-Merge and K-Merge++ methods were extensively tested across a wide range of real-world tasks, including Smart Reply, Summarization, Tone Adjustment, Question Answering, and Grammar Correction, spanning multiple languages. The results demonstrated the superiority of their approach compared to other strategies, all while adhering to the strict storage and computational limitations of on-device settings. For instance, the integration of a new LoRA takes mere milliseconds, proving its practicality for mobile deployment.
The research highlights that storing all individual LoRAs for 40 tasks would require about 1GB of storage, which is substantial for a mobile device. K-Merge and K-Merge++ can achieve about 80-90% of the performance of single-task LoRAs with just 8 merged clusters, significantly reducing the storage footprint. K-Merge++ particularly shines in its robustness to different task arrival orders, ensuring consistent performance even in less ideal scenarios.
This work represents a significant step forward in making powerful LLMs more practical and efficient for on-device deployment, allowing mobile devices to continually expand their language capabilities without compromising performance or exceeding storage limits.
