TLDR: A new study uses network analysis of imagination vividness ratings to compare internal world models (IWMs) in humans and large language models (LLMs). It reveals that human IWMs exhibit consistent structural similarities and distinct clustering patterns, reflecting a coherent mental organization. In contrast, LLM IWMs lack these human-like characteristics, showing inconsistent correlations and minimal clustering, suggesting fundamental differences in how they process and represent imagined experiences.
A groundbreaking study titled Internal World Models as Imagination Networks in Cognitive Agents by Saurabh Ranjan and Brian Odegaard from the University of Florida delves into the fundamental differences between human and artificial intelligence in how they imagine and construct their internal understanding of the world.
For a long time, imagination was thought to primarily help us maximize rewards. However, recent research challenges this idea. This new study proposes that imagination serves a deeper purpose: to access and utilize an ‘internal world model’ (IWM). To explore this, the researchers used a novel approach called psychological network analysis, comparing IWMs in humans and large language models (LLMs).
The Study’s Approach
The team assessed imagination vividness using two questionnaires: the Vividness of Visual Imagery Questionnaire (VVIQ-2) and the Plymouth Sensory Imagery Questionnaire (PSIQ). These questionnaires ask individuals to rate how vividly they can imagine various scenes or sensory experiences. From these ratings, ‘imagination networks’ were constructed. In these networks, each imagined scenario or item from the questionnaire became a ‘node,’ and the connections, or ‘edges,’ between them represented how strongly their vividness ratings were associated.
The study involved three human populations from Florida, Poland, and London, and several LLMs from the Gemma and Llama families. The LLMs were tested under two conditions: an ‘independent task’ where they responded to each item without memory of previous responses, and a ‘cumulative task’ where they retained conversational memory. To ensure a fair comparison, the researchers used a method called ‘population diversity sampling’ to make the LLM simulations reflect the natural variability in human imagination abilities.
Key Findings: A Tale of Two Imaginations
The results revealed striking differences between human and LLM internal world models. Human imagination networks consistently showed strong correlations between different ‘centrality measures’—metrics that indicate the importance of a node within a network. These included expected influence, strength, and closeness, suggesting that human internal world models are structurally similar across different populations. This means that for humans, the importance of one imagined scenario is consistently related to others, forming a coherent mental structure.
However, LLM imagination networks told a different story. They exhibited a lack of consistent clustering and generally lower correlations between centrality measures, regardless of the prompts or conversational memory conditions. While LLMs could report vividness ratings, these ratings varied significantly based on the instructions given about imagination ability (e.g., aphantasia, hyperphantasia), indicating that their ‘vividness’ is more a function of linguistic instruction than a stable internal experience.
One particularly interesting finding was related to ‘clustering.’ Human networks displayed clear, characteristic clusters that reflected the different contexts or sensory modalities of the questionnaire items. For example, visual items might cluster together. In contrast, most LLM networks in the independent task condition often formed only a single cluster, indicating a lack of distinct organizational structure. Even in the cumulative task, where some LLMs showed more than one cluster, their clustering alignment with human networks remained significantly lower.
Also Read:
- Beyond Mimicry: Rethinking AI’s ‘Theory of Mind’ and the Path to Mutual Understanding
- HA VIR: Reconstructing Complex Visual Scenes from Brain Activity with Hierarchical Processing
Implications for AI and Human Cognition
These findings suggest that while LLMs can process and generate language related to imagination, they do not organize their internal world models in a way that mirrors human experience. The human ability to form vivid mental images and experiences, influenced by long-term memories and phenomenological aspects, appears to result in a distinct internal structure that LLMs currently lack.
The researchers propose that imagination’s primary role might be to access a ‘recovery map’—an agent’s internal model of how the environment changes after actions—rather than solely for reward maximization. The consistent similarities in human imagination networks across populations could stem from shared recovery maps of common scenarios, while the differences in LLMs highlight their distinct underlying mechanisms.
This study provides a crucial framework for comparing internally-generated representations in humans and AI. It underscores that developing truly human-like imagination in artificial intelligence will require more than just linguistic capacity; it will necessitate a deeper understanding and replication of the complex phenomenological structures that underpin human internal world models.
