TLDR: A new research paper by Hao Ban and Kaiyi Ji re-evaluates parameter sharing in Low-Rank Adaptation (LoRA) for LLM fine-tuning. They discover that the similarity of LoRA’s ‘A’ matrices is due to initialization, not shared knowledge, and that the ‘B’ matrix is crucial for encoding domain knowledge. Based on this, they propose ALoRA for multi-task learning and Fed-ALoRA for federated learning, both sharing the ‘B’ matrix. These methods achieve more balanced performance, comparable or superior accuracy, and significantly reduce communication costs (up to 75% in federated settings) compared to previous approaches that shared ‘A’ matrices.
Large Language Models (LLMs) have become incredibly powerful, but adapting them for specific tasks, a process known as fine-tuning, can be very resource-intensive. Parameter-Efficient Fine-Tuning (PEFT) methods, like Low-Rank Adaptation (LoRA), offer a solution by only updating a small fraction of the model’s parameters, making the process more efficient.
LoRA works by representing weight updates as a product of two smaller matrices, ‘A’ and ‘B’. While a single LoRA can be effective, its capacity is limited when dealing with a wide variety of data or tasks. To overcome this, researchers often use multiple LoRAs, allowing each to specialize. However, simply adding more LoRAs increases computational and communication costs, especially in scenarios like multi-task learning (where one model handles several tasks) or federated learning (where multiple clients collaboratively train a model without sharing raw data).
Previous approaches to make multi-LoRA systems more efficient focused on sharing the ‘A’ matrices across different tasks or clients. The belief was that these ‘A’ matrices captured shared knowledge, leading to their observed similarity during training. Methods like HydraLoRA and FedSA-LoRA were built on this premise.
A New Perspective on Parameter Sharing
This new research challenges that assumption. By carefully re-examining the similarity of ‘A’ matrices, the authors found that this resemblance largely stems from their identical starting points (initialization), rather than from genuinely shared knowledge acquired during training. Their analysis revealed that the ‘A’ matrix primarily acts as a ‘feature projector,’ helping to transform input data into a useful format. In contrast, the ‘B’ matrix plays a much more critical role in ‘encoding domain knowledge’ – essentially learning the specific information relevant to a task or client.
Further experiments highlighted this distinction: the ‘A’ matrix remained largely unchanged throughout training, while the ‘B’ matrix showed significant adaptation in both its magnitude and direction. When ‘A’ matrices were shared, the learning process for ‘A’ was slow and prone to ‘gradient conflicts,’ hindering its ability to explore diverse feature patterns. Conversely, sharing ‘B’ matrices led to much more effective knowledge transfer, particularly in federated learning settings.
Introducing ALoRA and Fed-ALoRA
Motivated by these insights, the researchers propose two novel methods that flip the script: instead of sharing ‘A’, they share ‘B’.
ALoRA (Asymmetric LoRA) for Multi-Task Fine-Tuning: This architecture uses multiple ‘A’ matrices, each dynamically selected by an input-aware ‘router’, but shares a single ‘B’ matrix. This design allows individual ‘A’ matrices to explore different ways of projecting features, while the shared ‘B’ matrix acts as a central hub for fusing and transferring knowledge across tasks. This means the model can adapt to diverse tasks more effectively without sacrificing efficiency.
Fed-ALoRA for Federated Fine-Tuning: This method is designed for collaborative training across multiple clients. In Fed-ALoRA, only the ‘B’ matrices are communicated to and aggregated by a central server, significantly reducing communication costs.
- Homogeneous Setting: When all clients use LoRAs of the same size (rank), the shared ‘B’ matrices can be directly aggregated. This approach cuts communication costs by up to 50% compared to aggregating full LoRA parameters.
- Heterogeneous Setting: For clients with varying capacities and different LoRA ranks, direct aggregation of ‘B’ matrices would typically be impossible due to their differing sizes. Fed-ALoRA cleverly addresses this by decomposing the ‘B’ matrix into two components and introducing an auxiliary matrix for dimension adjustment. This allows for efficient aggregation even with diverse client setups, reducing communication costs by up to 75%.
Also Read:
- FLoRA-NA: Advancing Communication-Efficient and Accurate Federated Fine-Tuning for Large Language Models
- Dynamic Boosted Annealing: A New Approach to Efficient LLM Fine-Tuning
Demonstrated Effectiveness
The proposed ALoRA and Fed-ALoRA were rigorously tested across various benchmarks, including commonsense reasoning, math reasoning, multi-task NLP datasets, and federated NLP datasets. The results consistently showed that these new methods achieve more balanced performance across tasks and clients, with accuracy that is comparable to or even better than existing multi-LoRA approaches.
For instance, ALoRA outperformed HydraLoRA, a method that shares ‘A’ matrices, by improving average ROUGE-1 scores and achieving better performance balance. Similarly, Fed-ALoRA surpassed FedSA-LoRA in federated settings, demonstrating superior knowledge transfer and significantly reduced communication overhead.
This research provides a fresh perspective on parameter sharing in LLM fine-tuning, highlighting the distinct roles of ‘A’ and ‘B’ matrices in LoRA. By strategically sharing the ‘B’ matrix, ALoRA and Fed-ALoRA offer more efficient and effective ways to adapt LLMs for complex multi-task and federated learning environments. You can read the full paper here: Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs.
