TLDR: DivLogicEval is a new benchmark designed to more accurately evaluate the logical reasoning abilities of Large Language Models (LLMs). It features a dataset of natural sentences composed in counterintuitive ways to prevent reliance on common sense, and introduces a new metric, PartialCircular, for more faithful evaluation. Experiments show DivLogicEval effectively isolates logical reasoning, boasts high language diversity, and resists data contamination. Evaluations of current LLMs reveal significant limitations in their logical reasoning capabilities compared to humans, highlighting areas for future research and development.
Large Language Models (LLMs) have demonstrated impressive capabilities across various tasks, but accurately evaluating their logical reasoning skills remains a significant challenge. Traditional benchmarks often intertwine multiple reasoning abilities, such as common sense and logic, leading to an overestimation of an LLM’s pure logical prowess. Furthermore, many existing logical reasoning datasets suffer from limited language diversity and distribution biases, which can result in unreliable evaluations.
Addressing these critical issues, researchers have introduced a new benchmark called DivLogicEval. This innovative framework is specifically designed to provide a more faithful and robust evaluation of logical reasoning in LLMs. DivLogicEval stands out by focusing solely on classical logic, presenting natural language sentences in a deliberately counterintuitive manner to prevent models from relying on common sense or pre-trained shortcuts.
Building a Better Benchmark
The construction of DivLogicEval involves a meticulous two-step process. First, a verifiable classical logic expression is sampled. Then, this symbolic logic is transformed into natural language. This transformation is achieved by instantiating logical variables with diverse natural sentences, sourced from datasets like SNLI and MNLI, and then connecting them using counterintuitive sentence structures. To ensure high language quality and diversity, the process incorporates techniques like using varied sentence templates, instantiating the same variable with different sentences, and performing sentence negation. GPT-3.5 is also utilized for generating templates and conducting grammar checks, with manual review ensuring the final dataset’s integrity.
A More Reliable Evaluation Metric
Beyond the dataset, DivLogicEval also introduces a novel evaluation metric called PartialCircular. This metric aims to overcome the limitations of existing measures like ‘Accuracy’ and ‘Circular’, which can be inconsistent or fail to truly reflect a model’s confidence and reasoning ability. PartialCircular evaluates all possible permutations (mutants) of an instance’s options. It assigns a score that not only considers the number of correct predictions but also penalizes high entropy in the model’s predicted option distribution, effectively discouraging random guessing and providing a more nuanced assessment of confidence and correctness.
Why DivLogicEval is Effective
Experiments highlight DivLogicEval’s effectiveness in several ways. When LLMs were prompted to answer without using logical reasoning, their performance on DivLogicEval significantly decreased, unlike on other benchmarks where performance sometimes surprisingly improved. This indicates that DivLogicEval genuinely requires logical reasoning. The benchmark also boasts superior language diversity and a distribution closer to natural language compared to other classical logic datasets, mitigating distribution bias. Additionally, it demonstrates strong resistance to data contamination, a growing concern where LLMs might simply memorize answers from their training data rather than reason.
Also Read:
- Beyond Single Scores: How DeCE Evaluates LLMs for Accuracy and Completeness
- CultureScope: A Deeper Look into AI’s Cultural Competence
LLMs Still Have Room to Grow
A comprehensive evaluation of various LLMs, including Mixtral, LLaMA 3.3, Qwen 2.5, Gemini, GPT-3.5, GPT-4, GPT-4o, and o1-preview, revealed interesting insights. While the o1-preview model showed the best performance, its accuracy of 51.3% in a zero-shot setting still indicates substantial room for improvement in logical reasoning. Most other LLMs achieved accuracies below 36%, suggesting only a basic level of understanding. Notably, GPT-4’s performance surprisingly declined in a few-shot setting when presented with natural language DivLogicEval, though it benefited from few-shot examples in a symbolic version of the benchmark. This suggests that the counterintuitive nature of the language poses a unique challenge for some models. In stark contrast, human participants achieved an average accuracy of 86.7%, significantly outperforming all LLMs. This research underscores the current limitations of LLMs in complex logical reasoning and provides a valuable tool for future advancements in this critical area. For more details, you can refer to the full research paper here.
