TLDR: UrzaGPT is a new AI model that uses LoRA fine-tuning on large language models (LLMs) to improve card selection in Magic: The Gathering drafts. By training on human gameplay data, UrzaGPT enables even smaller LLMs to achieve competitive accuracy in predicting optimal card picks, demonstrating the viability of LLM-based agents for complex collectible card games and offering a path for adaptable, update-friendly AI in the future.
Collectible Card Games (CCGs) like Magic: The Gathering present unique challenges for artificial intelligence. Unlike traditional board games where AI has achieved superhuman feats, CCGs involve partial information, complex long-term strategies, and constantly evolving card sets. This complexity has meant that AI models often fall short of human expert performance in tasks such as deckbuilding and gameplay.
Introducing UrzaGPT: A New Approach to Card Drafting
A recent research paper introduces UrzaGPT, an innovative AI model designed to assist players with real-time card selection during the ‘drafting’ phase of Magic: The Gathering. Drafting is a crucial part of the game where players sequentially pick cards from packs to build their deck, a process that requires deep strategic understanding and adaptability.
UrzaGPT leverages the power of Large Language Models (LLMs), which are inherently well-suited for processing the natural language found on Magic: The Gathering cards. The researchers started with an open-weight LLM and fine-tuned it using a technique called Low-Rank Adaptation (LoRA) on a vast dataset of human draft logs. This approach allows the model to quickly adapt to the specific nuances of the game and even new card expansions.
How UrzaGPT Learns to Draft
The core idea behind UrzaGPT is that LLMs, with their language understanding capabilities, can learn to evaluate and select cards. Since simulating the full game to test drafted decks is difficult, the model was trained by predicting human decisions from a large dataset provided by 17lands.com, a platform where players record their games. The task for UrzaGPT was to predict which card a human player would pick, given the cards currently in their deck and the available cards in the pack.
An interesting finding was the choice of card representation. The study explored using just card names versus the full card text. Surprisingly, in initial tests, including the full card text slightly reduced performance. This suggests that LLMs might already have a strong understanding of cards just from their names, and too much additional text could introduce unnecessary noise.
Performance and Potential
The research benchmarked UrzaGPT against both untuned LLMs and existing specialized models. Smaller LLMs (like Llama-3-8B and Mistral-7B-Instruct) initially struggled to draft effectively without any specific training. However, after fine-tuning with UrzaGPT’s method, these smaller models showed remarkable improvement, reaching an accuracy of up to 66.2% in predicting human card selections. This significantly outperforms the zero-shot (untuned) performance of even larger models like GPT-4o (which achieved 43%).
While UrzaGPT’s performance doesn’t quite match the current state-of-the-art domain-specific models (which achieve around 68% accuracy), the gap is remarkably small. This demonstrates that using LLMs for drafting is not only possible but can also be highly performant, general, and easy to update for new game expansions.
Also Read:
- Enhancing AI Collaboration: How Language Models Learn to Navigate Complex Social Games
- New AI Harness Unlocks Strategic Diplomacy Play for Any Large Language Model
Future Implications
The success of UrzaGPT highlights the potential of LLMs in complex strategic games. The ability to quickly adapt smaller, more computationally efficient LLMs to specific game tasks opens doors for developing low-cost AI agents and player aids. These tools could rapidly learn new game mechanics and card sets, providing valuable assistance to players as the game evolves.
The researchers acknowledge that while predicting human picks is a good start, future work could involve evaluating the quality of entire drafted decks through gameplay simulations. This would provide a more holistic measure of the AI’s strategic prowess. For more in-depth details, you can read the full research paper here.
