TLDR: A new study reveals that while small language models can achieve high performance on specific Theory of Mind (ToM) tasks through reinforcement learning, they fail to generalize this understanding to new, unseen social reasoning scenarios. This suggests they are “hacking” statistical patterns in the training data rather than developing a true, adaptable ToM capability, highlighting limitations in current AI training and evaluation methods for social intelligence.
Recent advancements in artificial intelligence, particularly with Large Language Models (LLMs), have sparked considerable interest in their ability to mimic complex human reasoning. A key area of investigation is whether these models can develop a ‘Theory of Mind’ (ToM) – the crucial human ability to understand and attribute mental states like beliefs, desires, and intentions to oneself and others. This capacity is fundamental to social intelligence and would represent a significant leap for AI.
While larger LLMs have shown some nascent ToM-like abilities, the question of whether smaller models can acquire a robust and generalizable ToM remains contentious. A new research paper, titled “Small LLMs Do Not Learn a Generalizable Theory of Mind via Reinforcement Learning,” delves into this very question. Authored by Sneheel Sarangi and Hanan Salam from NYU Abu Dhabi, the study investigates if small-scale LLMs can truly learn a flexible and transferable ToM capability through Reinforcement Learning with Verifiable Rewards (RLVR).
Reinforcement Learning (RL) has been a game-changer in LLM training, allowing models to be optimized for specific outcomes beyond just predicting the next word. RL with Verifiable Rewards (RLVR) is particularly effective because it uses clear, rule-based rewards, avoiding the ambiguity of human feedback. This method has successfully boosted logical and mathematical reasoning in models like DeepSeek-R1, leading to skills that generalize well to new problems. The core inquiry of this research was whether this RL-driven success in formal reasoning could be replicated for the more nuanced domain of social reasoning.
The Study’s Approach
The researchers conducted a systematic evaluation using a 7-billion parameter model (Qwen2.5-7B-Instruct). They trained the model on various combinations of three prominent ToM datasets: HiToM, ExploreToM, and FANToM. These datasets were chosen for their diversity in input formats, narrative styles, and ToM challenges. For instance, FANToM features naturalistic dialogue, HiToM uses structured stories for higher-order reasoning, and ExploreToM includes adversarially generated false-belief scenarios.
To rigorously test generalization, the models were evaluated on entirely held-out datasets, meaning data they had never seen during training. These included OpenToM, specific list-response tasks from FANToM, and fourth-order reasoning tasks from HiToM (which were explicitly excluded from training). The training process utilized the REINFORCE++ algorithm, optimizing for a reward function that combined a ‘format reward’ (for structured output) and a ‘correctness reward’ (for accurate answers).
Key Findings: Mastery Without Generalization
The study’s findings revealed a significant discrepancy: while RL training led to substantial performance improvements on the tasks the models were specifically trained on (in-distribution tasks), this mastery did not translate to generalizable ToM capabilities. For example, models trained on FANToM showed a remarkable 65% improvement, and HiToM-trained models improved by 35% on their respective datasets. This confirms RLVR’s effectiveness for task-specific optimization.
However, when these models were tested on the unseen OpenToM benchmark or the FANToM List answering tasks, their performance remained largely stagnant, showing no significant improvement over an untrained baseline. In some cases, performance even degraded on tasks not included in their training regimen. This suggests that the learned ‘skill’ was narrowly tied to the specific training data and failed to transfer to novel scenarios or task formats.
The Phenomenon of “Hacking”
Perhaps the most compelling evidence of this narrow learning came from the HiToM dataset analysis. The baseline model showed a predictable difficulty curve, performing worse on higher-order reasoning tasks (e.g., fourth-order beliefs). However, the RL-trained models, especially those trained on lower-order tasks, paradoxically performed best on the most complex, unseen fourth-order tasks. This ‘inverted difficulty curve’ strongly indicates that the models were not learning a genuine, abstract ToM capability but rather ‘hacking’ or exploiting statistical patterns and structural artifacts within the templated HiToM data that became more pronounced in higher-order examples.
Furthermore, the study showed that the learned skills were brittle even to minor changes in task format. A model that achieved over 90% accuracy on FANToM’s binary false-belief questions showed no meaningful improvement on the FANToM List tasks, despite both relying on the same conversational context. This highlights that the model learned a rigid input-output mapping rather than a flexible internal representation of mental states.
Also Read:
- Unpacking Inverse Scaling: How Longer AI Reasoning Can Reduce Accuracy
- AI’s Achilles’ Heel: Why More Feedback Can Harm Large Language Model Performance
Implications for AI Development
The research concludes that for small LLMs, applying RLVR to current ToM benchmarks does not lead to the emergence of a genuine, general-purpose Theory of Mind. The behaviors learned are narrow, brittle, and indicative of sophisticated pattern matching rather than abstract social intelligence. The training dynamics clearly showed overfitting, with in-distribution accuracy rising while out-of-distribution accuracy remained flat.
These results serve as a cautionary tale, underscoring the limitations of current evaluation paradigms and suggesting that high benchmark scores can be misleading. Developing truly socially intelligent AI will likely require advancements beyond merely optimizing for correct answers on existing benchmarks. This could involve more robust and diverse training data, or novel reward mechanisms that can assess the fidelity of the reasoning process itself, rather than just the final outcome. For more details, you can read the full research paper here.
