TLDR: A new benchmark, ‘Beyond One World,’ evaluates large language models (LLMs) on their ability to consistently role-play 90 versions of 30 iconic superheroes across Marvel and DC universes. The study uses two tasks: ‘Canon Events’ for factual recall and ‘Moral Dilemmas’ for ethical decision-making, assessing models’ ‘thinking’ (internal deliberation) and ‘acting’ (outward decisions). Key findings show that chain-of-thought prompting has mixed effects, cross-version generalization remains a major hurdle, and models often struggle to align their internal reasoning with character-faithful actions, highlighting critical gaps in multiversal consistency for role-playing LLMs.
Large language models, or LLMs, are becoming increasingly sophisticated at mimicking human-like conversation and even adopting specific personalities. This capability, known as character-based role-playing, allows these AI models to act as given personas, emulating their knowledge, speaking style, and behavior. However, a new study highlights a significant challenge: can these models consistently and accurately portray different versions of the same character, especially when those characters exist across multiple fictional universes?
Think about iconic superheroes like Spider-Man or Batman. Over decades of storytelling, these characters have appeared in countless comics, movies, and TV shows, each offering a slightly different take on their history, values, and moral codes. This rich, complex tapestry of character versions provides an ideal testing ground for evaluating the true depth of an LLM’s role-playing abilities.
Introducing the ‘Beyond One World’ Benchmark
A team of researchers has introduced a new benchmark called Beyond One World to explore this very problem. This benchmark is designed to assess how well LLMs can perform character-grounded role-play across multiversal contexts. It features 30 iconic heroes, each represented by 90 canon-specific versions, drawing from the vast Marvel and DC universes.
The benchmark includes two main tasks:
- Canon Events: This task tests the LLM’s factual recall of pivotal life stages for each character, such as their childhood, pre-hero phase, and established-hero phase. Models are presented with multiple-choice questions about key moments in a hero’s timeline.
- Moral Dilemmas: Here, models are confronted with ethically charged scenarios inspired by common superhero narrative themes. These dilemmas force the AI to make choices consistent with a character’s ethical code, exploring conflicts like ‘save one vs. the greater good’ or ‘duty vs. personal desire’.
To evaluate responses, the researchers developed a unique framework that separates a model’s internal deliberation (what they call “thinking”) from its outward decisions (“acting”). They also introduced a metric called “Think–Act Matching,” which quantifies how well a model’s stated reasons align with its chosen actions, serving as a proxy for trustworthiness in its role-play.
Also Read:
- Unlocking Human Intent: A New Benchmark for Evaluating Language Models in Real-World Consumer Discussions
- Crafting Authentic Game NPCs: How “Deflanderization” Balances Personality and Purpose
Key Findings from the Research
Experiments with various reasoning-oriented and standard LLMs revealed several interesting insights:
- Chain-of-Thought Prompting: While using chain-of-thought (CoT) prompting—where models explain their reasoning step-by-step—improved narrative coherence in weaker models, it surprisingly reduced canonical accuracy in stronger ones. This suggests that too much explicit reasoning can sometimes lead to models generating information that strays from the established canon.
- Cross-Version Generalization: A significant challenge identified was the models’ difficulty in generalizing across different versions of the same character. They often struggled to distinguish between overlapping but distinct timelines, indicating a major obstacle in achieving multiversal consistency.
- Thinking vs. Acting: The study found that models often excelled at either internal deliberation (“thinking”) or outward decision-making (“acting”), but rarely both. A model might articulate consistent reasoning but then act inconsistently with the character’s persona, or vice versa. Bridging this gap is crucial for creating truly trustworthy role-playing agents.
In essence, “Beyond One World” exposes critical limitations in current LLMs’ ability to maintain multiversal consistency and align their reasoning with character-faithful actions. These findings suggest that future advancements in role-playing LLMs will need to focus on more integrated reasoning and persona modeling, potentially by combining structured knowledge with dynamic narrative alignment to truly capture the nuances of complex characters across different stories.
