TLDR: This research paper proposes a three-stage framework to systematically evaluate the quality of AI-generated game narratives. It leverages a Delphi study with narrative design experts to validate key story quality dimensions and then maps these dimensions onto the Kano model to understand their impact on player satisfaction. The findings help game developers prioritize quality aspects when co-creating stories with AI, also identifying emergent crucial dimensions like ‘Voice’ and ‘Genre Alignment’.
The integration of Artificial Intelligence, particularly Large Language Models (LLMs), into video game development has opened new frontiers for creating dynamic and engaging narratives. From generating dialogue for Non-Player Characters (NPCs) to crafting entire story arcs and quests, AI offers unprecedented flexibility and personalization. However, this rapid advancement also brings a significant challenge: ensuring the consistent quality of these AI-generated narratives.
A recent research paper, “Evaluating Quality of Gaming Narratives Co-created with AI”, by Arturo Valdivia and Paolo Burelli from the IT University of Copenhagen, addresses this critical need by proposing a structured methodology to evaluate the quality of AI-generated game narratives. The authors highlight that while AI has a long tradition in procedural content generation for games, LLMs introduce a new level of sophistication, yet the quality of their output can be unpredictable, potentially harming player immersion and experience.
A Three-Stage Evaluation Framework
The paper introduces a comprehensive three-stage framework designed to systematically assess story quality. This framework aims to help game developers identify the most relevant story quality dimensions for LLM-generated narratives and anticipate how these dimensions influence player satisfaction.
The first stage involves compiling an initial list of “story quality dimensions” (SQDs) – variables that impact a story’s quality. This list was derived from existing literature, identifying twenty-three common dimensions used for story evaluation.
The second stage focuses on validating and refining this list through a Delphi study. The Delphi method is an iterative process that gathers and distills knowledge from a panel of experts using rounds of questionnaires and controlled feedback. For this study, a panel of ten experts in narrative design for games and technical practitioners working with LLM-generated stories participated. These experts came from diverse backgrounds, including AAA studios, free-to-play titles, and indie developers, ensuring a broad range of perspectives.
The third stage classifies these validated quality dimensions using the Kano model framework. Originally used for customer requirement analysis, the Kano model helps understand how the presence or absence of specific attributes affects satisfaction. In this context, it helps categorize SQDs based on their impact on player satisfaction into categories like “Delighter” (unexpected positive features), “Performance” (satisfaction proportional to presence), “Must-have” (basic expectations, absence causes dissatisfaction), “Indifferent” (neither increases nor decreases satisfaction), and “Reverse” (causes dissatisfaction when present).
Also Read:
- Navigating the Ethical Landscape of Generative AI: A New Evaluation Framework
- AI Agents Tackle Full Storylines in Classic Adventure Games
Key Findings and Emergent Dimensions
The initial round of the Delphi study yielded significant insights. None of the twenty-three initial SQDs were deemed unimportant, with 78% receiving a median importance score of at least 3.5 (considered “Very important”). When mapped to the Kano model, over half (57%) of the SQDs were classified as “Performance” (or One-dimensional), meaning player satisfaction is directly proportional to how well the story performs in these areas. Additionally, 26% were classified as “Must-haves,” and 13% as “Delighters” (or Attractive).
Crucially, the expert panel also identified two new, vital dimensions not initially captured: “Voice” and “Genre Alignment.” Experts noted that while “Naturalness” was an existing dimension, it didn’t fully encompass the nuanced aspects of a distinctive and compelling narrative voice, which is a critical marker of high-quality storytelling. Similarly, genre alignment was highlighted as fundamental, as a story failing to adhere to or meaningfully engage with its genre conventions can be a significant quality failure, regardless of technical competence.
By incorporating these emergent dimensions, the evaluative framework becomes more robust, better capturing the nuanced assessments of narrative experts. This research provides valuable guidance for game developers on prioritizing quality aspects when co-creating game narratives with generative AI, ultimately aiming to enhance player experience and immersion.
