TLDR: A research paper evaluates Large Language Models (LLMs) on metaphor interpretation, finding that their performance is primarily driven by surface features like lexical overlap and sentence length, not deep understanding. LLMs surprisingly perform better on original metaphorical datasets (due to higher lexical overlap) than on automatically generated literal paraphrases, highlighting the need for more realistic evaluation methods for figurative language.
A recent research paper delves into the intriguing question of how Large Language Models (LLMs) interpret metaphorical language. Titled “Metaphor and Large Language Models: When Surface Features Matter More than Deep Understanding,” this study by Elisa Sanchez-Bayona and Rodrigo Agerri from the HiTZ Center – Ixa at the University of the Basque Country UPV/EHU offers a comprehensive evaluation of LLMs’ capabilities in understanding metaphors.
Previous research in Natural Language Processing (NLP) on metaphor interpretation has often been limited. Many studies relied on single datasets or specific task settings, frequently using artificially constructed data where literal words were simply replaced with metaphorical ones. This approach, as highlighted by Sanchez-Bayona and Agerri, can introduce biases, particularly lexical overlap, which might skew the results.
To address these limitations, the researchers conducted extensive experiments using a diverse range of publicly available datasets. These datasets included inference and metaphor annotations, and the evaluation focused on Natural Language Inference (NLI) and Question Answering (QA) tasks. The core questions the study aimed to answer were: Does the presence of metaphors impact LLMs’ ability to make correct inferences? Do LLMs show generalization or emergent abilities in understanding metaphorical language across different tasks, prompts, and datasets? And do lexically replaced datasets introduce biases that explain high LLM performance?
The study’s findings are quite insightful. The results indicate that the performance of LLMs in metaphor interpretation is more significantly influenced by surface-level features like lexical overlap (how many words are shared between sentences) and sentence length, rather than a deep understanding of the metaphorical content itself. This suggests that any perceived “emergent abilities” of LLMs to grasp metaphorical language might actually stem from a combination of these surface-level features, in-context learning (where the model learns from examples provided in the prompt), and its general linguistic knowledge.
The researchers evaluated several prominent LLMs, including Llama-3-8B-Instruct, Llama-3.3-70B-Instruct, Mistral-7B-Instruct, Qwen/Qwen2.5-7B-Instruct, Qwen2.5-72B-Instruct, gemma-3-4b-it, and gemma-3-27b-it. They tested various prompt configurations, including zero-shot (no examples), few-shot (a few examples), and Chain-of-Thought (CoT) prompting, which provides a more detailed explanation of the task. Interestingly, few-shot and CoT setups consistently outperformed fine-tuned encoder models, reducing the need for large amounts of manually annotated data.
A particularly revealing part of the study involved evaluating LLMs on “adversarial literal paraphrases.” These were automatically generated literal versions of the metaphorical sentences. Surprisingly, LLMs often performed better on the original datasets containing metaphors than on their literal paraphrases. The analysis revealed that the original metaphorical datasets, often created through lexical substitution, exhibited a higher lexical overlap between premises and hypotheses. This higher overlap, along with shorter sentence lengths, correlated with better performance. Conversely, longer sentences and lower lexical overlap in the literal paraphrases led to poorer performance.
The manual error analysis further illuminated these points. Errors in the literal paraphrases often occurred because the paraphrased sentences still contained metaphors, or the paraphrase altered the original meaning, leading to a “label shift.” Another common issue was a decrease in lexical overlap between the premise and hypothesis in the literal versions, making it harder for the models to establish the inference.
Also Read:
- Unpacking How Question Types Affect Large Language Model Performance
- Unpacking AI’s Social Smarts: Why Small Language Models Struggle with Generalizable Theory of Mind
In conclusion, this research provides critical insights into the current capabilities and limitations of LLMs when processing figurative language. It underscores the importance of developing more realistic evaluation frameworks for metaphor interpretation tasks. The paper emphasizes that while LLMs show strong performance, it’s often due to their sensitivity to surface-level features and in-context learning rather than a true deep understanding of metaphor. For more details, you can read the full research paper here.
