TLDR: A study compared 15 Large Language Models (LLMs) with human participants on 8 custom logical and abstract reasoning questions. While LLMs showed strengths in some pattern recognition and calculations, they struggled with nuanced understanding, abstract thought, and unconventional problem formats, revealing a gap in higher-order cognitive functions compared to humans, especially professors.
The field of Artificial Intelligence has seen remarkable advancements with Large Language Models (LLMs) demonstrating impressive capabilities in natural language processing and text generation. However, a crucial question remains: can these models truly reason at a human level? A recent study titled Human-Level Reasoning? A Comparative Study of Large Language Models on Logical and Abstract Reasoning by Benjamin Grando Moreira delves into this very question, comparing the logical and abstract reasoning skills of various LLMs against human performance.
The Core Challenge: Beyond Language
Evaluating LLMs’ reasoning ability is vital because it goes beyond merely processing language. It assesses whether these models genuinely understand information, can make inferences, and draw logical conclusions. The study highlights that despite their proficiency in tasks like translation and content creation, LLMs often fall short in challenges requiring deeper logical and critical thinking. Previous research suggests that while LLMs excel at capturing statistical patterns in language, they frequently show inconsistencies when applying logical principles.
Study Design and Methodology
This comparative analysis involved fifteen distinct LLMs, including well-known models like GPT, Claude, DeepSeek, Gemini, Grok, Llama, Mistral, Perplexity, and Sabi´a. Each LLM was presented with an initial prompt to solve logical and abstract challenges and explain their reasoning process. This approach aimed to evaluate not just the correctness of answers but also the validity of the underlying thought process.
The researchers developed eight custom-designed reasoning questions, specifically crafted to minimize reliance on factual knowledge and instead emphasize logical or abstract pattern identification. These questions covered various reasoning types: deductive (e.g., day of the week operations, Caesar cipher, order of operations), inductive (month name translation), and abductive (binary number operations). Two questions were specifically designed to assess abstract reasoning, involving a pattern of word repetitions and a complex numerical sequence based on month properties.
For comparison, the same set of questions was administered to 80 human participants, comprising students and professors from quantitative disciplines. Responses from both LLMs and humans were qualitatively categorized as correct answer, correct reasoning despite an incorrect answer, or no answer, with a scoring system assigning points accordingly. Unlike some studies, this research did not use formal metrics like Perplexity or BLEU, focusing instead on a detailed analysis of the content of the answers.
Key Findings: LLMs vs. Humans
The study revealed significant differences in performance. The average score for LLMs was 59.7 points, with GPTo4-mini, GPTo3-mini, and Claude 3.7 Sonnet achieving the highest scores. Conversely, Sabi´a 3, Llama 3.1, Gemini 2.0 Flash, and Grok 2 showed the poorest performance.
LLMs demonstrated proficiency in certain tasks, such as correctly identifying that none of the provided options were correct in the order of operations question (Question 4) and associating abbreviated month names with their Spanish equivalents (Question 6). However, they frequently struggled with questions requiring nuanced understanding and abstract thought. For instance, the complex numerical sequence in Question 5, which involved concatenating the square of the month’s position with the number of letters in its Portuguese name, proved particularly challenging for most LLMs. Similarly, Question 7, which involved operations with days of the week but incorporated an alternating language pattern (Portuguese or English) based on whether the increment was even or odd, also posed difficulties for LLMs in discerning the language switch rule.
Human participants, especially professors, generally outperformed LLMs on tasks requiring abstract reasoning and the application of common-sense knowledge. While the aggregate performance of LLMs (73.4%) was slightly higher than humans (69.6%) overall, humans exhibited superior performance on several individual questions, particularly those demanding abstract reasoning and the ability to adapt to unconventional problem formats.
Also Read:
- Unmasking the Limits of Large Reasoning Models: A Deep Dive into Complexity
- AI’s Journey to Solve Math Word Problems: A Cognitive Perspective
The Reasoning Gap
The findings underscore a persistent gap between artificial and human intelligence, particularly in higher-order cognitive functions. While LLMs have made considerable strides in pattern recognition and language processing, their ability to truly “reason” appears limited by their reliance on statistical correlations rather than genuine understanding. They often struggled with unconventional problem formats, such as deliberately omitted correct answers or hidden rules for language switching.
Future research, as suggested by the authors, should explore the impact of different prompting strategies, including Chain-of-Thought reasoning, which was not uniformly applied in this study. This could provide further insights into how to enhance the reasoning capabilities of LLMs and bridge the existing gap with human cognitive abilities.
