TLDR: Researchers used Sparse Autoencoders (SAEs) with a GPT-style model trained on Jane Austen’s novels to understand how LLMs internalize social concepts and biases from their training data. They found that the model’s internal representations clearly reflected themes like gender, marriage, and class, demonstrating how SAEs can act as a scalable tool for corpus exploration and bias discovery in LLMs.
Large Language Models (LLMs) are trained on vast amounts of data, making it incredibly challenging to understand what they truly learn and how biases from their training data are internalized. A recent study introduces a novel approach to tackle this problem by combining LLMs with Sparse Autoencoders (SAEs), offering a window into the deeper structures, themes, and biases embedded within the training data.
The researchers trained a GPT-style transformer model exclusively on the complete novels of Jane Austen. This specific corpus was chosen because it is rich in social constructs and narrative patterns, providing a controlled environment to observe how an AI internalizes complex societal themes like gender, class, and duty. By using a smaller, well-understood dataset, the study aimed to isolate the effects of a specific cultural and historical context on the model’s learned representations.
Sparse Autoencoders (SAEs) are a key component of this method. They work by mapping complex neural activations within the LLM into a sparse latent space. This process helps to disentangle overlapping concepts that might be compressed into single neurons, making the underlying features more interpretable. In this study, SAEs were applied to the hidden states across multiple layers of the GPT-style model, revealing sparse, interpretable features that directly reflected the key narratives and concepts present in Austen’s novels.
The analysis revealed fascinating insights into how the model encoded social concepts. High-confidence neurons were strongly associated with gendered concepts, with many neurons consistently activating for ‘female’ or ‘male’ related sentences. This suggests that gender roles are among the most distinct and robustly encoded features. Additionally, concepts like ‘marriage’ and ‘wealth’ frequently co-activated, reflecting their intertwined nature in Austen’s narratives, where marriage often served as a strategic alliance influenced by financial considerations.
Further investigation into ‘dual-theme neurons’ showed how concepts interact within the model’s internal representations. In earlier layers (Layer 6), concepts began to group into modular clusters, with ‘marriage’ and ‘wealth’ forming bridges between different domains. As the model processed information deeper (Layer 12), the conceptual space became more densely interconnected. ‘Male’ emerged as a central theme, linking to a majority of other concepts, suggesting masculinity serves as a semantic anchor in the model’s deeper understanding. ‘Emotion’ and ‘duty’ also formed a strong pairing, while ‘society’, initially peripheral, became fully integrated, closely entangled with ‘love’, ‘marriage’, ‘female’, ‘wealth’, and especially ‘male’. Interestingly, ‘class’ remained relatively isolated, primarily connected only to ‘wealth’, mirroring Austen’s implicit treatment of class through financial status.
A concept-level similarity analysis further illustrated these shifts. In Layer 6, all themes showed high cosine similarity, indicating a densely entangled conceptual space where social concepts were richly overlapping. By Layer 12, the model displayed a more differentiated and structured conceptual space, with themes arranged into more distinct semantic groupings. For instance, ‘wealth’ and ‘class’ formed a strong structural pairing, while ’emotion’ and ‘love’ were closely aligned affective themes. ‘Family’ became more isolated from emotional themes, aligning more with ‘marriage’ and ‘female’, reflecting a shift towards a structural or gendered role rather than an emotional one.
Also Read:
- Guiding Language Models: How Sparse Autoencoders Enhance Reasoning
- Unlocking the Black Box: A New Way to Understand How LLMs Think
These findings demonstrate that LLMs combined with SAEs can act as scalable probes into complex datasets. This approach offers a new path for corpus exploration, bias discovery, and model interpretability at scale. By treating the model not just as an object of interpretation but as a tool for dataset analysis, researchers can uncover how hidden social representations are embedded in the training data itself, paving the way for identifying and mitigating bias at its source. For more details, you can read the full research paper here.
