TLDR: A new study investigates whether Large Language Models (LLMs) learn grammar in a graded, meaning-infused way, similar to human language acquisition theories. By analyzing Pythia-1.4B’s internal representations of English dative constructions, researchers found that LLMs create distinct geometric clusters for constructions based on human preference strength. This indicates that LLMs learn rich, graded representations that align with usage-based constructionist principles, suggesting a sophisticated understanding of linguistic nuances beyond simple classification.
Language is a complex system, and how humans learn and represent it has been a subject of extensive research. One prominent theory, the usage-based constructionist (UCx) approach, suggests that language is not a rigid set of rules but rather a dynamic network of learned form-meaning pairings, known as constructions. A key aspect of this approach is that grammaticality isn’t a simple yes or no, but a continuum of acceptability, influenced by meaning and context.
Recent advancements in Artificial Intelligence, particularly with Large Language Models (LLMs), have opened new avenues to explore these linguistic theories. LLMs are powerful learners, trained on vast amounts of text data, making them ideal candidates to investigate whether they spontaneously develop representations that mirror the meaning-infused, probabilistic, and graded nature proposed by the UCx framework.
Investigating LLMs’ Internal Grammar
A recent study, titled Meaning-infused grammar: Gradient Acceptability Shapes the Geometric Representations of Constructions in LLMs, delved into this very question. Conducted by Supantho Rakshit and Adele E. Goldberg from Princeton University, the research focused on how LLMs represent English dative constructions, specifically the Double Object (DO) construction (e.g., “She gave the boy the book”) and its alternative, the Prepositional Object (PO) construction (e.g., “She gave the book to the boy”). The choice between these two is known to be heavily influenced by semantic and pragmatic factors.
The researchers used Pythia-1.4B, a publicly available LLM, and a dataset called DAIS (Dative Alternation and Information Structure). This dataset is unique because it contains 5,000 sentence pairs, each conveying the same core meaning but presented in both DO and PO forms, along with human-collected acceptability judgments. These human judgments allowed the researchers to categorize sentences into five tiers based on how strongly humans preferred one construction over the other, ranging from “Top 10% Strongest Preference” to an “Unbiased Baseline” (where preferences were ambiguous).
Unveiling Geometric Representations
Instead of just looking at whether the LLM could correctly classify constructions, the study focused on the “underlying geometry” of the LLM’s internal representations. They analyzed the neural activations (hidden states) of Pythia-1.4B for each sentence. To measure the separability of these representations, they employed two sophisticated statistical measures: Energy Distance and Jensen-Shannon Divergence. These measures help quantify how distinct entire clouds of representations are from each other in a high-dimensional space.
The findings were compelling and consistent: the geometric distance between the representations of DO and PO constructions was systematically modulated by the gradient of human preference strength. In simpler terms, when humans had a strong preference for one construction over the other, the LLM’s internal representations for those constructions were more distinct and separated in its activation space. Conversely, for sentences where human preference was ambiguous, the representations were less distinct and more overlapping.
This pattern held true across nearly all layers of the Pythia-1.4B model, and importantly, the researchers confirmed these findings were robust across the entire Pythia model suite, from smaller 70M parameter models to larger 6.9B parameter models. This suggests that the observed phenomenon is a general property of transformer architectures, not just specific to one model size.
Also Read:
- Unveiling LLM’s Inner Workings: How “Curved Inference” Maps Semantic Shifts
- Unpacking LLM Intelligence: A New Look at How Models Process Information
Implications for Language Understanding
These results provide strong computational evidence supporting a core principle of the usage-based constructionist approach: that grammatical representations are graded and sensitive to semantic and pragmatic context. It demonstrates that LLMs develop internal representations whose geometric properties are highly compatible with the probabilistic, usage-based categories posited by this linguistic theory.
The study extends previous work that primarily focused on an LLM’s ability to classify constructions, by showing a more fine-grained, graded geometric structure within the model. It also aligns with other research indicating that LLMs learn subtle linguistic nuances, such as which verbs are less likely to appear in certain constructions based on usage patterns. Interestingly, the effects observed were strongest in the mid-layers of the LLM, suggesting that these rich, graded geometric representations are most clearly formed as part of a mid-level process of semantic abstraction within the model.
In conclusion, this research bridges the gap between theoretical linguistics and modern Natural Language Processing. It suggests that LLMs learn a rich, dynamic, and meaning-infused model of grammar that is highly compatible with a constructionist perspective, offering exciting new directions for understanding how these powerful AI models process and represent human language.
