TLDR: A new research paper explores how well Large Language Models (LLMs) and Large Reasoning Models (LRMs) align with human judgments of mathematical problem interestingness and difficulty. The study found that while many LLMs broadly correlate with human notions of interestingness, they often fail to capture the full distribution of human judgments or the specific reasons why humans find certain problems engaging, such as elegance. Mistral models showed the strongest alignment. The findings highlight both the potential and current limitations of AI in understanding subjective mathematical appeal.
Mathematics, at its heart, is not just about solving problems, but also about choosing which problems are worth solving. This fundamental human aspect of mathematical exploration—the judgment of a problem’s ‘interestingness’ and ‘difficulty’—is now being explored in the realm of artificial intelligence. A recent study delves into how well large language models (LLMs) and large reasoning models (LRMs) align with human perceptions of what makes a math problem engaging.
The research, titled “A Matter of Interest: Understanding Interestingness of Math Problems in Humans and Language Models,” highlights that while AI systems have made incredible strides in solving complex mathematical problems, their ability to autonomously judge the inherent value or appeal of a problem remains largely unexplored. This capability is crucial for future applications, from AI tutors suggesting engaging problems to students, to AI systems making new mathematical discoveries.
To investigate this, the researchers conducted two empirical studies. The first involved participants from a crowdsourcing platform who rated problems from the American Mathematics Competitions (AMC). The second study engaged International Math Olympiad (IMO) competitors, who assessed past IMO problems. Both groups rated problems for interestingness and difficulty on a scale of 0-100 and provided rationales for their judgments. The study then compared these human judgments with those from 12 different language models across five families.
The findings reveal a nuanced picture. On average, LLM judgments of interestingness and difficulty showed a correlation with human judgments. Notably, models from the Mistral family demonstrated the strongest agreement with human perceptions. This suggests that current LLMs can approximate human notions of interestingness with surprising fidelity.
However, a significant divergence appeared when looking at the *distribution* of interestingness judgments. While LLMs might agree on which problems are generally interesting, they often don’t capture the full range and variability of human responses. This indicates that while models can identify interesting problems, they might not reflect the diverse ways humans perceive and value them.
When it came to *why* humans found problems interesting, elegance emerged as a key factor for IMO problems. Participants frequently cited reasons like “the problem statement is simple and elegant,” “the solution does not require any sophisticated techniques/theorems,” and “the solution is elegant.” Most LLMs, however, did not reflect these human-selected rationales well, with only the Mistral 7B Instruct and Mistral 24B Instruct models showing better alignment.
An interesting observation regarding Large Reasoning Models (LRMs) was their “judgment speed.” For simpler problems (like those in the crowdsourced study), LRMs tended to make quicker, “flash” judgments for problems they deemed uninteresting, while investing longer reasoning chains (more computational resources) for problems they found interesting. This distinction, however, disappeared for the more challenging Olympiad-style problems, where the effort to simply understand the problem likely overshadowed the effort related to judging its interest.
Also Read:
- Enhancing LLM Reliability: Why Step-by-Step Confidence Matters in Complex Tasks
- Enhancing LLM Confidence: A New Approach Using Reasoning Path Geometry and Uncertainty Analysis
This research represents an important first step in understanding the alignment between human and AI judgments of mathematical interestingness. The authors acknowledge limitations, such as focusing on competition math problems and specific participant populations, which might not fully represent the breadth of mathematical experience. Nevertheless, the study opens doors for future work, particularly in developing AI systems that can serve as more effective mathematical thought partners, capable of understanding and adapting to human notions of interestingness across different levels of expertise. You can read the full paper here.
