TLDR: A new research paper investigates the ability of Large Language Models (LLMs) to understand word senses in context. The study evaluates LLMs on Word Sense Disambiguation (WSD) tasks, comparing them to specialized systems, and also assesses their performance in generative tasks like definition and example creation. Key findings show that top LLMs like GPT-4o and DeepSeek-V3 achieve performance on par with state-of-the-art WSD systems and demonstrate high accuracy (up to 98%) in explaining word meanings in free-form generative settings, showcasing greater robustness across domains. However, LLMs still lag behind human performance, especially with challenging and infrequent word senses.
Large Language Models (LLMs) have become incredibly powerful, driving advancements across many natural language processing applications. From generating text to answering complex questions, these models seem to understand language remarkably well. However, a fundamental question remains: do LLMs truly grasp the nuanced meanings of words in different contexts? A recent research paper delves into this very topic, exploring how LLMs handle lexical ambiguity, a common feature of human language.
The study, titled “Do Large Language Models Understand Word Senses?”, was conducted by a team of researchers including Domenico Meconi, Simone Stirpe, Federico Martelli, Leonardo Lavalle, and Roberto Navigli. Their work addresses a critical gap in understanding the capabilities of LLMs beyond their impressive performance in various generative tasks.
Evaluating Word Sense Disambiguation (WSD)
One of the primary ways the researchers investigated LLM understanding was through Word Sense Disambiguation (WSD). This task involves identifying the correct meaning of a word when it appears in a specific sentence, often by choosing from a list of dictionary definitions. The team evaluated a wide range of instruction-tuned LLMs, from smaller models with a few billion parameters to very large ones like GPT-4o and DeepSeek-V3, comparing their performance against state-of-the-art WSD systems specifically designed for this task.
The findings were quite illuminating. Leading LLMs such as GPT-4o and DeepSeek-V3 achieved performance levels comparable to, and in some cases even surpassing, specialized WSD systems like ConSeC and ESCHER. This suggests that these advanced LLMs are highly capable of selecting the correct word sense from a predefined inventory. Furthermore, LLMs demonstrated greater robustness across different datasets, including those designed to be particularly challenging or diverse in domain, where specialized WSD systems often struggled.
Interestingly, the study also explored the impact of prompt design and context. While adding more context didn’t always lead to significant improvements, shuffling the order of candidate definitions revealed a potential limitation: smaller LLMs sometimes exhibited a positional bias, favoring earlier options rather than purely semantic appropriateness. This highlights the sensitivity of LLMs to how questions are phrased and options are presented.
Despite their strong performance, LLMs are not yet on par with human-level WSD. An expert human annotator achieved a significantly higher F1 score compared to GPT-4o on a subset of the data, indicating that there’s still a gap in truly nuanced understanding, especially with difficult or less frequent word senses.
Lexical Understanding Through Generation
Beyond simply selecting a definition, the researchers wanted to see if LLMs could demonstrate their understanding in more unconstrained, generative settings. They tasked Llama-3.3-70B-Instruct and GPT-4o with three types of generation tasks:
- Definition Generation: Creating a dictionary-style definition for a word in context.
- Free-form Explanation: Explaining the meaning of a word in context in their own words.
- Example Generation: Producing three new sentences that use the target word with the same meaning as in a given sentence.
These generative tasks were evaluated by human experts. The results were impressive, especially for free-form explanations, where LLMs achieved up to 98% accuracy. This suggests that when LLMs are allowed to express their understanding freely, they can demonstrate a profound grasp of word senses. In definition generation, most outputs were judged to be of similar quality to gold standard definitions, with GPT-4o generally outperforming Llama-3.3-70B-Instruct.
For example generation, models successfully created relevant sentences 60-70% of the time. However, a notable portion of these examples showed a repetitive structural pattern, indicating that while the semantic understanding was there, the models sometimes lacked syntactic variety in their generated usages.
Also Read:
- Unmasking AI’s Shallow Grasp of Puns: New Research Reveals LLMs Struggle with True Humor Understanding
- Navigating Ambiguity: A Benchmark for Named Entity Recognition
Common Errors and Future Directions
The analysis of errors revealed several patterns, including overgeneralization (selecting an overly broad sense), metonymy (confusing closely related senses), and gross misinterpretation (selecting a sense clearly divergent from the context). These insights are crucial for further improving LLM capabilities.
In conclusion, this comprehensive study provides strong evidence that top-performing LLMs like GPT-4o and DeepSeek-V3 are highly capable in Word Sense Disambiguation, often matching or exceeding specialized systems and showing greater robustness across diverse contexts. Their ability to explain word senses in generative settings, particularly in free-form explanations, is remarkably high. While they still trail human performance in the most challenging cases, these findings underscore the significant progress in LLM understanding of lexical ambiguity. For more details, you can read the full research paper here.
