TLDR: This research uses “crosscoders” to track how interpretable features (concepts) emerge and evolve within large language models during pre-training. It identifies two phases: an early statistical learning phase and a later feature learning phase where complex concepts form. The study links specific feature development to improvements in downstream task performance, making the LLM training process more transparent.
Large Language Models (LLMs) have become incredibly powerful, but how they acquire their vast capabilities during pre-training has largely remained a mystery. Researchers often refer to this process as a “black box.” A new study sheds light on this opaque process by tracking the evolution of internal concepts within these models as they learn.
The research, titled “Evolution of Concepts in Language Model Pre-Training,” introduces a novel approach using a method called “crosscoders.” Traditionally, methods like sparse autoencoders (SAEs) have been used to identify interpretable features in fully trained models. However, these methods didn’t show how these features develop over time. Crosscoders, a variation of SAEs, are adapted in this work to simultaneously analyze activations from different stages of a model’s training. This allows scientists to observe where these internal features emerge, change, and even fade away.
Think of it like watching a child learn: you don’t just see the final adult, but you track their development from infancy through adolescence. Similarly, crosscoders enable tracking the “development” of concepts within an LLM. A key insight from this method is that the “decoder norms” of these crosscoders directly indicate the strength and presence of a particular feature at any given training snapshot.
Unveiling Feature Evolution Patterns
The study revealed two primary patterns in how features evolve:
- Initialization Features: These are features that exist from the very beginning of training, often showing an initial drop and recovery before gradually decaying. These are like innate predispositions.
- Emergent Features: Most features, however, begin to form around a specific point in training (roughly step 1000 in the Pythia models studied). They then grow in intensity, reaching their peak at various later stages. These are the learned concepts.
Interestingly, the research found a correlation between when a feature emerges and its complexity. Simpler features tend to appear earlier, while more complex patterns, such as understanding sentence structures or nuanced contexts, emerge in later training stages. For example, features related to the “previous token” (what came right before) appear early, while “induction features” (recognizing repeating patterns) and “context-sensitive features” appear later.
Another significant observation was a “universal directional turning point” around step 1,000. Before this point, feature directions are almost orthogonal to their directions after this point, suggesting a major reorientation in how the model represents information early in its learning journey.
Connecting Internal Features to Model Behavior
Beyond just observing feature evolution, the researchers also established a causal link between these microscopic feature changes and the model’s macroscopic performance on various tasks. Using attribution-based circuit tracing techniques, they investigated how the formation of specific features impacts downstream task metrics. For instance, they studied tasks like subject-verb agreement, induction (predicting repeating patterns), and indirect object identification.
By selectively removing or isolating these identified features, the team could consistently disrupt or recover the model’s performance on these tasks across different training stages. This confirms that the crosscoders successfully pinpoint the necessary and sufficient components for a model to perform a given task.
Also Read:
- Unlocking the Internal Mechanics of Language Model Fine-Tuning
- Decoding LLM Learning: A Network Perspective on Transformer Dynamics
Two Phases of Learning
The study proposes a compelling two-phase learning process during LLM pre-training:
- Statistical Learning Phase: In the very early stages of training, the model rapidly decreases its loss by learning coarse statistical patterns, such as the frequency of single words (unigrams) and pairs of words (bigrams). During this phase, internal representations are dense.
- Feature Learning Phase: After the initial statistical learning is largely complete, sparse, distinct features begin to emerge in “superposition” (where multiple concepts are encoded in overlapping ways). This phase is crucial for further loss reduction and the development of more complex capabilities.
This two-stage structure aligns with existing theories like the information bottleneck theory, which describes an initial “fitting” phase followed by a “compression” phase.
This groundbreaking work, detailed further in the full research paper, opens up exciting possibilities for understanding the fine-grained representation progress during language model learning. By making the “black box” a little more transparent, it paves the way for developing more robust, interpretable, and controllable LLMs in the future.
