TLDR: A new automated pipeline uses large language models (LLMs) to efficiently estimate Verb Frame Frequencies (VFFs), which describe how often verbs appear in specific grammatical structures. This method generates a large corpus of sentences and then uses an LLM to analyze their syntax, outperforming traditional parsing tools and providing a scalable, accurate way to understand verb usage in language.
Understanding how verbs are used in language, specifically the grammatical structures they appear in, is crucial for both human language comprehension and the development of artificial intelligence. This concept is known as Verb Frame Frequencies (VFFs). While word frequency is well-studied, VFFs have been less explored due to the significant challenges in accurately calculating them.
Traditional methods for determining VFFs often involve manual analysis by trained linguists, which is highly accurate but incredibly time-consuming and resource-intensive. Automated tools exist, but they frequently suffer from inaccuracies, especially with complex sentence structures or informal language. This has limited the scale and accessibility of high-quality VFF data.
A new research paper, titled A Scalable Pipeline for Estimating Verb Frame Frequencies Using Large Language Models, introduces an innovative, fully automated pipeline that leverages the power of Large Language Models (LLMs) to overcome these limitations. The pipeline, developed by Adam M. Morgan and Adeen Flinker, offers a fast, accurate, and scalable solution for estimating VFFs.
How the New Pipeline Works
The process begins by selecting a comprehensive list of English verbs. For this study, 476 verbs were chosen, including those from existing gold-standard datasets and verbs relevant to specific linguistic phenomena like dative and locative alternations. Next, an LLM (specifically, OpenAI’s GPT-o1) is used to generate a thousand diverse contexts, such as “on the tennis court” or “down to the wire.”
Then, another LLM (GPT-4o-mini) is employed to generate sentences. For each of the 476 verbs, 100 sentences are created, resulting in a large corpus of over 45,000 sentences. The LLM is instructed to produce natural-sounding sentences with varied meanings, tenses, and syntactic structures, mimicking real-world language use.
The most critical step is parsing, where the syntactic structure of each generated sentence is analyzed. Instead of relying solely on traditional automated parsers like the Berkeley Neural Parser or Stanford CoreNLP, this pipeline uses a powerful LLM (GPT-4o) instructed to act as an expert linguist. This LLM identifies the verb’s arguments and tags them using standard linguistic labels, effectively converting the sentence corpus into a ‘treebank’ – a collection of sentences annotated with their syntactic structures.
Superior Performance and Key Advantages
The researchers rigorously evaluated their LLM-based parsing method against established benchmarks and traditional parsers. The results were compelling: the GPT-4o parser consistently outperformed both the Berkeley Neural Parser and the Stanford CoreNLP parser across multiple evaluation datasets. For instance, when compared to the gold-standard Gahl et al. (2004) dataset, the GPT-4o VFFs accounted for significantly more variance in verb frame distributions.
Furthermore, human validation confirmed the LLM parser’s accuracy. An expert linguist manually annotated a subset of sentences, and the GPT-4o parser showed a 79% agreement rate with these human annotations, notably higher than the 69% for Berkeley and 59% for Stanford. The pipeline also demonstrated its ability to accurately capture subtle verb biases, such as those involved in NP/SC (Noun Phrase/Sentential Complement) ambiguities and dative alternations, aligning closely with human norming studies.
A significant improvement offered by this pipeline is its more fine-grained syntactic distinctions. Unlike some previous datasets that collapsed categories like intransitive and prepositional phrase (PP) frames, this new approach treats selected PPs as distinct arguments, providing a more accurate and psycholinguistically relevant understanding of verb usage.
Also Read:
- How AI Models Learn Grammar: A Deep Dive into Meaning-Infused Language Representations
- AI Breakthrough: Generating High-Quality Old English Texts
Future Implications
This automated pipeline represents a significant leap forward in VFF estimation. By making the process fast and scalable, it opens doors for extensive research in cognitive science and Natural Language Processing (NLP). The resulting VFF database is more comprehensive, covering a broader range of verbs and structural alternations with finer distinctions than existing resources.
The methodology is highly customizable and extensible, meaning it can be adapted to analyze new verbs, different syntactic frames, and even other languages. As LLMs continue to evolve, this pipeline could enable rapid, domain-specific estimation of syntactic preferences, leading to advancements in psycholinguistic modeling and improved syntactic generalization in neural network architectures.
While the study acknowledges limitations, such as the potential for biases inherited from the LLM’s training data and the need for further exploration of model parameters, it stands as a robust proof of concept for automated verb frame frequency estimation, providing valuable tools and data for the research community.
