TLDR: A study compared human and 31 large language model (LLM) comprehension of seven complex sentence structures, including ‘garden path’ sentences, double center-embedded sentences, depth-charge sentences, and similarity-based interference sentences. It found LLMs struggle similarly to humans, particularly with ‘garden path’ sentences, where even strong models like GPT-5 showed significant performance drops. The similarity in difficulty ranking between humans and LLMs increased with model size. Additionally, LLMs exhibited human-like performance gaps between difficult and simple sentences (directionality) only when they were in a ‘sweet spot’ – neither too weak nor too strong. The research suggests that while LLMs’ larger working memory aids in some complex structures, garden path sentences, requiring reanalysis, pose a distinct challenge.
Large language models (LLMs) are increasingly capable of fluent human-like conversation, but do they experience the same linguistic processing difficulties as humans? A recent research paper, “Comparing human and language models sentence processing difficulties on complex structures,” delves into this question, systematically comparing how humans and five families of state-of-the-art LLMs comprehend challenging linguistic structures. The study, conducted by Samuel Joseph Amouyal, Aya Meltzer-Asscher, and Jonathan Berant, provides fascinating insights into the similarities and differences in sentence processing between artificial intelligence and human cognition.
The researchers set out to bridge gaps in previous studies by directly measuring sentence comprehension in both humans and LLMs using a unified experimental framework. They focused on seven distinct types of complex sentence structures, each designed to challenge different aspects of cognitive processing. These included four types of “garden path” (GP) sentences, which initially mislead the reader into an incorrect interpretation, requiring reanalysis. Other structures examined were double center-embedded sentences (stressing working memory), depth-charge sentences (involving multiple negations), and similarity-based interference sentences (where similar noun phrases cause confusion).
For each structure, the study created pairs of sentences: a “target” sentence with the difficult structure and a “baseline” sentence where the difficulty was neutralized. Both humans and 31 different LLMs (varying in size and training) were tasked with answering a comprehension question after reading each sentence. Human performance validated that these structures are indeed challenging, with an average accuracy of 28.3% across tasks.
The findings reveal that LLMs, while generally performing better than humans, still struggle significantly with these complex structures. The highest mean accuracy for LLMs was 74.5% without ‘thinking’ tokens and 88.9% with ‘thinking’ for the strongest models, indicating that these sentences are challenging for AI as well. A particularly striking observation was that LLMs struggled most with garden path (GP) sentences. For instance, GPT-5 achieved near-perfect accuracy (93.7%) on non-GP structures but plummeted to 46.8% on GP structures. This suggests a fundamental difference in how LLMs handle reanalysis compared to other forms of linguistic complexity.
Interestingly, the study found that the rank correlation between human and LLM performance on the difficulty of structures increased with the LLM’s parameter count, meaning larger models tend to rank the difficulty of different sentence types more similarly to humans. The model ‘o4-mini’ showed the highest correlation at 0.93. Furthermore, the researchers observed a “sweet spot” for LLM performance: models that were neither too weak nor too strong exhibited human-like “directionality,” meaning their accuracy on target (difficult) sentences was lower than on baseline (simple) sentences. Very weak models performed poorly on both, while very strong models performed equally well on both, effectively bypassing the human-like difficulty gap.
The influence of “thinking” tokens (a mechanism where models generate internal reasoning steps) also varied. While thinking generally improved performance for stronger models, it was less effective, and sometimes even detrimental, for GP structures. This leads to a conjecture that GP sentences, which require discarding a wrong interpretation, pose a different kind of challenge than structures that primarily tax working memory, where LLMs have a significant advantage due to their larger memory capacity.
Also Read:
- Unpacking Metaphorical Minds: How LLMs Grapple with Figurative Language
- Unpacking LLM Memory: New Research Reveals Early Forgetting in Complex Reasoning Tasks
In conclusion, this research highlights both convergence and divergence in how humans and LLMs process complex sentences. While LLMs demonstrate human-like difficulties, especially with garden path sentences, their superior working memory allows them to outperform humans on other memory-intensive structures. The “sweet spot” rule for human-like error patterns offers a valuable framework for understanding the nuanced similarities between artificial and human language comprehension. For more details, you can read the full paper here.
