AI for Game Design: Automating Monster Level Estimation in Pathfinder RPG

TLDR: A master’s thesis explores using predictive machine learning, specifically ordinal regression, to automatically estimate monster challenge levels in pen&paper RPGs, focusing on Pathfinder Second Edition. The research developed a dataset of 2,637 monsters and evaluated various machine learning models against a human-inspired baseline. Tree-based models like Random Forest and LightGBM consistently outperformed the baseline and many neural network approaches, achieving high accuracy (often within one level of the true value). This work demonstrates the potential of AI to significantly shorten game development cycles and assist Game Masters in creating balanced custom encounters.

The world of pen&paper role-playing games (RPGs) has seen remarkable growth in recent years, with the market projected to reach over $5 billion USD by 2033. As this industry expands, game publishers are increasingly looking towards artificial intelligence (AI) to enhance player experiences and maintain a competitive edge. One significant challenge in game design, particularly for Game Masters (GMs) and developers, is accurately determining the challenge level of new opponents, often referred to as monsters.

Traditionally, estimating a monster’s difficulty level relies heavily on manual testing and expert evaluation. These methods, while capable of providing accurate assessments, are notoriously time-consuming and resource-intensive. Imagine spending countless hours in playtesting sessions, or manually comparing a new monster to thousands of existing ones to gauge its power. Even official guidelines, such as those provided by Pathfinder Second Edition, can be imprecise, especially for complex creatures with unique abilities.

A recent master’s thesis, titled Application of predictive machine learning in pen&paper RPG game design, explores how predictive machine learning can automate this crucial aspect of game design. Authored by Jolanta Śliwa, with supervision from PhD Wojciech Czech and Jakub Adamczyk, this research focuses specifically on the popular Pathfinder Second Edition system.

The Machine Learning Approach to Monster Levels

The core idea behind this research is to treat monster level estimation as an ‘ordinal regression’ problem. Ordinal regression is a type of machine learning that predicts a categorical outcome where the categories have a natural order (like ‘low’, ‘medium’, ‘high’ difficulty), but the exact difference between each category isn’t necessarily uniform. For Pathfinder, monster levels range from -1 to 21, representing a clear progression in power.

To achieve this, a unique dataset was created, comprising 2,637 distinct monster entries from official Pathfinder rulebooks. Each monster was described by 32 features, including core attributes like Strength, Dexterity, and Intelligence, defensive capabilities such as Armor Class (AC) and Hit Points (HP), saving throws (Fortitude, Reflex, Will), movement speeds (land, fly, swim), number of immunities, and detailed spellcasting and attack statistics (spell count per level, spell difficulty, attack bonuses, average damage).

Evaluating Different Models

The study evaluated a wide array of machine learning models, categorized into two main groups:

1. Classical Regression Models with Rounding: These models, including Linear Regression (like LASSO), Support Vector Machines (SVM), Random Forests (RF), and Light Gradient Boosting Machine (LightGBM), predict a continuous numerical value. This value is then ’rounded’ to the nearest integer level using various strategies, from simple mathematical rounding to more sophisticated methods that optimize thresholds between levels.
2. Dedicated Ordinal Regression Models: These algorithms are specifically designed to handle ordered categorical data directly. Examples include Simple Approach to Ordinal Classification (SAOC), Ordered Logistic Regression (ORD), Gaussian Processes for Ordinal Regression (GPOR), Ordered Random Forest (ORF), and several deep neural network (DNN) approaches (CORAL, CORN, Spacecutter, NNRank, CONDOR, OR-CNN).

A ‘human-inspired’ baseline model was also developed, mimicking how a game designer might estimate a monster’s level by assigning an archetype and comparing it to similar creatures. This served as a benchmark for the machine learning models.

Key Findings and Performance

The evaluation methodology was particularly robust, using a ‘chronological train-test split’ and an ‘expanding window’ approach. This simulated real-world scenarios where models predict levels for newly published monsters based on older data, preventing data leakage and providing a more realistic assessment of generalization.

The results were compelling:

• The human-inspired baseline model performed significantly worse than all machine learning approaches, highlighting the complexity of manual level estimation.
• Tree-based models, such as Random Forest, LightGBM, Ordered Random Forest (ORF), and Simple Approach to Ordinal Classification (SAOC) with a Random Forest base, consistently emerged as the top performers across various metrics. They demonstrated strong generalization capabilities and robustness.
• Interestingly, there wasn’t a clear overall winner between classical regression models with rounding and dedicated ordinal regression models. Both categories contained high-performing algorithms.
• Many deep neural network models for ordinal regression, especially those that didn’t explicitly enforce rank consistency, generally underperformed compared to the tree-based methods. More advanced neural network techniques showed improvement but still lagged behind the best tree-based models.
• A crucial finding was that most machine learning models achieved high ‘accuracy@1’ (over 95%). This means that even when a model didn’t predict the exact monster level, it was usually off by only one level, indicating a high degree of practical usefulness.
• The choice of rounding strategy for classical regression models was important, with graph-based threshold tuning often yielding the best results, though simple mathematical rounding was more reliable for models prone to overfitting, like LightGBM.

Also Read:

• A New Arena for AI Coders: Benchmarking Goal-Oriented Software Development
• Estimating Model Performance with Synthetic Data: A New Approach for Data-Scarce Environments

Future Directions and Practical Applications

This research demonstrates that machine learning can reliably estimate monster levels, opening up exciting possibilities for the RPG industry. Future work could involve refining rounding threshold optimization using dedicated validation sets and exploring more advanced loss functions for neural networks.

The practical applications are immediate and impactful:

• Accelerated Development: Game designers could significantly shorten the development cycle for new monsters by using these models as an initial estimator, reducing the need for extensive, time-consuming playtesting.
• Empowering Game Masters: GMs who create custom monsters for their campaigns often lack the resources for thorough balancing. Integrating these models into a user-friendly application could provide GMs with a powerful tool to design balanced and challenging encounters, enhancing the overall player experience.

A prototype application based on this work has already shown promise in improving session balance, underscoring the real-world value of applying AI to the creative and complex domain of tabletop RPG game design.