TLDR: A research paper investigates how Large Language Models (LLMs) compare entities based on numerical attributes. It reveals that LLMs often ignore their own accurate numerical knowledge, instead relying on heuristic biases like entity popularity, mention order, and semantic co-occurrence. Larger models are found to be more strategic, using numerical knowledge when reliable, while smaller models do not discriminate. Chain-of-Thought prompting improves reliance on numerical features but doesn’t eliminate all inconsistencies, suggesting a hybrid reasoning approach in LLMs.
Large Language Models (LLMs) are becoming increasingly central to tasks that require knowledge-based reasoning. However, a significant challenge remains: understanding when these powerful AI models genuinely apply their internal knowledge versus when they fall back on simpler, superficial shortcuts. A recent research paper titled “Knowing the Facts but Choosing the Shortcut: Understanding How Large Language Models Compare Entities” delves into this very question, offering crucial insights into the decision-making processes of LLMs.
The researchers investigated this phenomenon by tasking LLMs with comparing entities based on numerical attributes, such as “Which river is longer, the Danube or the Nile?” This setup provides a clear, objective correct answer, allowing for systematic analysis. Surprisingly, the study found that even when LLMs possessed the correct numerical information, they frequently made predictions that contradicted this knowledge.
The study identified three primary heuristic biases that significantly influence LLM predictions:
Entity Popularity
Models tend to favor entities that are more popular, assuming they might also have higher values for a given attribute. For example, a more famous city might be incorrectly assumed to have a larger population, even if its actual population is smaller.
Mention Order
The order in which entities are presented in a prompt can bias the model’s choice. LLMs often show a preference for the first or second entity mentioned, regardless of the actual facts.
Also Read:
- Unpacking Forgetting in Language Models: A Deep Dive into Post-Training Knowledge Retention
- Unpacking Bias in AI’s Thought Process: How Language Models Aggregate Stereotypes
Semantic Co-occurrence
Shallow statistical associations between words can mislead models. If an entity frequently appears alongside adjectives like “tallest” or “largest” in its training data, the model might associate it with high values, even if its actual numerical attribute is lower.
To rigorously test these biases, the researchers designed a “Balanced-Orthogonal Subset” of test questions. This careful design ensured that each bias could be isolated and measured independently, preventing confounding factors. The findings revealed that mention order was the most dominant shortcut for all models. Popularity and co-occurrence also consistently influenced predictions, though to a lesser extent.
A critical discovery was the difference in behavior between models of varying sizes. Smaller LLMs (7–8 billion parameters) showed no discrimination, frequently relying on these shortcuts even when they possessed more accurate numerical knowledge. In contrast, larger models (32 billion parameters) demonstrated a more strategic approach: they selectively relied on numerical knowledge when it was more reliable and accurate. This explains why larger models often outperform smaller ones, not necessarily because they have more accurate knowledge, but because they are better at deciding when to use it.
The study also explored the impact of Chain-of-Thought (CoT) prompting, a technique that encourages models to verbalize their reasoning process. CoT prompting was found to steer all models, regardless of size, towards using numerical features more consistently. This led to an improvement in both pairwise accuracy and internal consistency, meaning the models’ predictions aligned better with their own numerical understanding. However, the researchers noted that CoT reasoning didn’t completely eliminate inconsistencies, as models sometimes rationalized their answers or retrieved inaccurate numbers during the “thinking” process.
In conclusion, this research offers a nuanced perspective on how LLMs operate. Rather than being purely “world models” that understand facts or “statistical parrots” that merely mimic patterns, LLMs appear to function in a hybrid manner. They possess genuine knowledge but don’t always deploy it consistently. The ability to strategically choose between knowledge-driven reasoning and heuristic shortcuts is a capability that emerges with increased model scale, suggesting a form of meta-cognitive development in larger LLMs. This work provides a foundational step towards a more sophisticated understanding of LLM behavior, shifting the focus from whether models understand to when and how they choose different reasoning strategies. You can read the full paper for more details: Knowing the Facts but Choosing the Shortcut.
