TLDR: A new study reveals that online reinforcement learning (RL) fine-tuning significantly alters the internal workings of large language models (LLMs). It increases the strength and number of active internal connections (activation intensity) and diversifies how these connections are used (activation patterns). This makes information flow more redundant and flexible, potentially explaining RL’s advantage in generalization. In contrast, DPO-based RL fine-tuning showed weaker or inconsistent internal changes, suggesting that dynamic online interaction is key to these internal enhancements.
Large language models, or LLMs, have become incredibly powerful, largely due to extensive pretraining and further refinement through methods like supervised fine-tuning (SFT) and reinforcement learning (RL)–based post-training. While it’s clear that RL fine-tuning often makes LLMs even better than SFT alone, the exact reasons why this happens deep inside the models have remained a mystery.
A new study dives into this unexplored territory, drawing inspiration from a technique called edge attribution patching (EAP) to examine the internal workings of LLMs before and after RL fine-tuning. The researchers wanted to understand how RL fine-tuning, which has been shown to improve LLM capabilities in tasks like writing, reasoning, and coding, actually changes the model’s internal information flow.
Key Findings: Stronger Signals and Diverse Pathways
The study found two consistent and significant effects of online RL post-training across several different LLM families. Firstly, there was an overall increase in “activation intensity.” This means that more internal pathways within the LLM were engaged, and the signals traveling through these pathways became stronger. Imagine it like more roads opening up in a city, and traffic flowing more robustly on all of them.
Secondly, the researchers observed greater “diversity in activation patterns.” This was reflected by higher entropy and less concentrated distributions of how different internal connections were used. In simpler terms, the model started using its internal connections in a more varied and flexible way, rather than relying on a few dominant pathways. This suggests that RL makes the information flow both more robust (redundant) and adaptable (flexible), which could explain why RL-tuned models are better at generalizing to new tasks.
The DPO Exception
Interestingly, models fine-tuned with Direct Preference Optimization (DPO) showed different results. They exhibited much weaker or inconsistent internal changes compared to models trained with PPO (Proximal Policy Optimization) and GRPO (Group Relative Policy Optimization). The researchers explain this by noting that DPO, unlike PPO and GRPO, uses a fixed dataset for training, similar to SFT. This means DPO doesn’t allow for the continuous regeneration of responses and application of reward signals to evolving policies, which is a hallmark of true online RL. This distinction highlights that the dynamic, interactive nature of online RL is crucial for reshaping the LLM’s internal dynamics in these specific ways.
Also Read:
- Decoding LLM Reasoning: How Reinforcement Learning and Supervised Fine-Tuning Shape Thought Processes
- Unpacking How Large Language Models Specialize for Rare Words
How the Research Was Conducted
To conduct this research, the team used an efficient Edge Attribution Patching (EAP) framework. This involved treating the LLM’s internal residual pathways as a directed graph, where each sub-module (like an attention block or feed-forward block) is a node, and the information flow between them are edges. They quantified the importance of these internal connections by estimating loss perturbations using gradient-based linearization, a method that is computationally efficient for large models. They also carefully selected and truncated model-generated token sequences to ensure fair and comparable analysis.
The experiments involved four pairs of open-source LLMs, each with a base model and an RL-tuned counterpart, tested on mathematical benchmarks like GSM8K, MATH, and College Math. The metrics used were Activation Intensity (average magnitude of edge weights), Information Complexity (Shannon entropy of edge activations), and Distribution Kurtosis (shape and stability of edge-weight distributions).
The findings provide a unified understanding of how RL fine-tuning systematically alters the internal structure of LLMs, bridging the gap between observed performance gains and the underlying changes in how information is processed. This research offers valuable insights for the future development of both LLMs and their post-training methodologies. For more technical details, you can read the full paper here: Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs.
