TLDR: Reinforcement learning (RL) fine-tuning of large language models (LLMs) surprisingly updates only a small fraction (5-30%) of parameters, leaving most unchanged. This sparse subnetwork is consistent across different RL algorithms and datasets. Training only this identified subnetwork can fully recover the performance of a densely fine-tuned model, suggesting RL acts as a precise, targeted adjustment rather than a broad overhaul. This discovery has implications for more efficient LLM training and understanding how models adapt.
Reinforcement learning (RL) has become a crucial step in refining large language models (LLMs), helping them to perform complex tasks and align with human preferences. Traditionally, it was thought that achieving new behaviors through RL fine-tuning required adjusting most of the model’s internal settings, or parameters.
However, a recent research paper challenges this long-held assumption with surprising findings. The study reveals that RL fine-tuning actually modifies only a small portion of an LLM’s parameters—typically between 5% and 30%. This means the vast majority of the model’s internal weights remain essentially untouched. This phenomenon is termed “RL-induced parameter update sparsity,” and it occurs naturally, without any special techniques designed to enforce sparsity.
This sparsity was consistently observed across a wide range of RL algorithms, including popular ones like PPO, DPO, and PRIME, and across various LLM families from different developers. In contrast, standard supervised fine-tuning (SFT) tends to update a much larger proportion of the model’s parameters, often leaving only 5% to 15% unchanged. This suggests that RL fine-tuning is far more targeted in its adjustments.
Interestingly, the specific subnetwork of parameters updated by RL is not random. The research found significant overlap in these updated subnetworks even when different random seeds, training datasets, or even different RL algorithms were used. This consistency is much greater than what would be expected by chance, indicating that the pre-trained model might have a partially transferable structure—a set of “knobs” that RL reliably turns to achieve alignment.
A key finding, formalized as Conjecture 1 in the paper, proposes that if this specific subnetwork of updated parameters is identified, one can fine-tune only that subnetwork (while keeping all other weights frozen) and achieve a model nearly identical to the one fine-tuned with the full model. Experiments supported this conjecture, showing that training only the identified subnetwork recovered, and in some cases even slightly exceeded, the performance of the fully fine-tuned model. Furthermore, the resulting model parameters were almost indistinguishable from the fully fine-tuned version, with over 99.9% of parameters matching.
The paper also delves into why RL updates only a sparse subnetwork. The primary factor appears to be the nature of the RL training data. RL fine-tuning often occurs on data that is very similar to the model’s own outputs, or close to its pre-trained distribution. This means the model only needs to make small, targeted adjustments to its behavior. In contrast, supervised fine-tuning often involves data that diverges more significantly from the model’s current generation style, necessitating more extensive parameter updates.
This research deepens our understanding of how RL-driven model adaptation works. It implies that RL concentrates its training efforts on a small, consistently active subnetwork, leaving most of the model’s weights unchanged. This sheds new light on why RL fine-tuning tends to preserve pre-trained capabilities better than supervised fine-tuning. The findings also open doors for more efficient RL fine-tuning methods that could exploit this intrinsic update sparsity, for example, by focusing computational resources only on the identified subnetwork. This also provides a new perspective on the “lottery ticket hypothesis” in the context of LLM alignment.
Also Read:
- Exploring How Different Data Domains Influence AI Reasoning in Language Models
- Optimizing Large Language Model Training Through Dynamic Data Weighting
For more in-depth details, you can read the full research paper available here.
