TLDR: A study reveals that large language models develop internal “modal difference vectors” that allow them to distinguish between probable, improbable, impossible, and inconceivable events more reliably than previously thought. These internal representations evolve with model development and closely align with human judgments, suggesting LMs encode a nuanced understanding of reality, potentially offering insights into human cognition.
Language models, or LMs, are incredibly versatile tools, capable of everything from answering complex questions to crafting imaginative stories. But for them to be truly reliable, especially in critical applications, they need to understand the difference between what’s real, what’s possible, what’s impossible, and what’s just plain nonsensical. Recent studies have raised questions about their ability to make these distinctions, known as modal categorization.
A new research paper, titled “Is This Just Fantasy? Language Model Representations Reflect Human Judgments of Event Plausibility,” by Michael A. Lepori, Jennifer Hu, Ishita Dasgupta, Roma Patel, Thomas Serre, and Ellie Pavlick, delves into this very challenge. Their work uncovers how LMs internally represent these different categories of events, offering a fresh perspective on their understanding of the world.
Unveiling Internal Representations
The researchers identified what they call “modal difference vectors” within various language models. These are essentially linear representations in the LM’s hidden state space that capture the difference between modal categories. Think of them as internal compasses that point the LM towards understanding whether a sentence describes something probable, improbable, impossible, or inconceivable.
Unlike previous methods that relied on a sentence’s overall probability (how likely the LM predicts the next word), these modal difference vectors proved to be far more accurate and reliable in classifying sentences by their modal category. This suggests that LMs possess a deeper, more structured understanding of plausibility than just surface-level probability estimates.
How LMs Learn Reality
The study also explored how these modal understanding capabilities develop within LMs. They found that the ability to distinguish between modal categories emerges gradually, much like in human development. Simpler distinctions, such as separating completely nonsensical statements from everything else, appear earlier in the model’s training, in shallower layers, and in smaller models. More nuanced distinctions, like telling the difference between improbable and impossible events, develop later as the models become larger and more sophisticated.
Aligning with Human Intuition
Perhaps one of the most fascinating findings is how closely these LM representations align with human judgments. The researchers modeled human categorization behavior, which often involves graded, intuitive notions rather than strict, expert-defined labels. They discovered that the feature space derived from these modal difference vectors accurately reflected how human participants categorized scenarios. This means that when a scenario falls somewhere between two categories, LMs exhibit a similar level of uncertainty or disagreement as humans do.
What Drives LM Judgments?
To understand what features LMs use to make these distinctions, the team correlated the modal difference vectors with human ratings of various interpretable features, such as subjective event likelihood, imageability (how easy a sentence is to visualize), grammaticality, and whether a sentence makes sense. They found that the vector distinguishing probable from improbable events strongly correlated with subjective event likelihood.
Most notably, the vector differentiating impossible from inconceivable events correlated selectively with features related to visualization and the presence of physical objects or environments. This suggests that an LM’s ability to “imagine” a scenario might be crucial in distinguishing between something that violates physical laws (impossible) and something that is fundamentally nonsensical (inconceivable). This insight could even inform our understanding of how humans make these distinctions.
Also Read:
- The Internal Battle: How Large Language Models Decide Between Facts and Contextual Information
- Early Warning for AI Safety: Monitoring Language Model Thoughts
Looking Ahead
This research provides compelling evidence that language models learn to represent the difference between reality and fantasy to a greater extent than previously believed. These findings open up exciting avenues for future research, such as using modal difference vectors to test an LM’s intuitive theories about the world, similar to how we study human understanding of physics. It also raises the intriguing possibility of using LM representations to generate new hypotheses about human cognition, particularly regarding the role of mental imagery in understanding different types of events. For more details, you can read the full paper here.
