TLDR: A study explores how large language models (LLMs) can act as cooperative agents in multiplayer UNO. Researchers found that while all tested LLMs could play UNO better than a random agent, only the largest model (LLaMA3.3-70B) significantly improved a teammate’s win rate in cooperative play, especially with “cloze prompting.” The study highlights that model size and prompting methods both influence an LLM’s effectiveness as a strategic partner.
Large Language Models (LLMs) are rapidly evolving beyond simple question-answering tools, moving towards taking direct, agentic actions to support desired outcomes. However, their role as non-player characters (NPCs) actively assisting a human player’s strategic goals in competitive environments has remained largely unexplored. A recent research paper delves into this very question, investigating whether an LLM-based agent can effectively help another player achieve their strategic goal as an active participant in a game.
The researchers, Yago Romano Martinez and Jesse Roberts from Tennessee Tech University, chose UNO, a popular turn-based card game, as their testing ground. Crucially, the LLM’s objective was not to win the game itself, but to aid another player in securing victory. To facilitate this, they developed a specialized tool that allows decoder-only LLMs to participate as agents within the RLCard game environment, a platform designed for reinforcement learning research in card games.
Evaluating LLM Performance in UNO
The study evaluated a range of decoder-only LLMs, from smaller models with 1 billion parameters (LLaMa3.2-1B) to much larger ones with 70 billion parameters (LLaMA3.3-70B). The investigation focused on two primary settings:
- Autonomous Play: Here, the LLM competed head-to-head against a random agent to establish a baseline for its competence.
- Cooperative Play: In this scenario, the LLM was paired with a rule-based teammate in a three-player game, with the goal of improving its partner’s win rate.
The research also explored how factors like model scale and different prompting techniques—specifically ‘cloze prompting’ and ‘counterfactual prompting’—influenced the LLM’s performance in both autonomous and cooperative roles.
Key Findings: LLMs as UNO Players and Teammates
In autonomous play, all tested LLMs successfully outperformed a random baseline, demonstrating their ability to make strategic decisions in UNO. Notably, the Mistral-24B model achieved the highest win rate of 52.10% using cloze prompting, showing that even without specific fine-tuning, these LLMs can exhibit strong general reasoning capabilities in structured decision-making environments.
The more challenging aspect was cooperative play. While all models could play UNO, only a few were able to significantly aid another player. The LLaMA3.3-70B model, when utilizing cloze prompting, delivered the most substantial assistance gain, being the only configuration to achieve statistical significance in improving its teammate’s win rate. This suggests that while smaller models can play, larger models show a greater capacity for effective cooperation.
Regarding model scale, the findings indicated that while a larger size generally helps, it’s not the sole determinant of success. The smallest model, LLaMA3.2-1B, sometimes rivaled or even surpassed larger models in certain scenarios, implying that the specific architecture and the prompting technique employed can sometimes outweigh sheer parameter count.
The study also shed light on prompting techniques. Cloze prompting generally yielded more consistent performance gains across various models in both solo and collaborative settings, suggesting it might be a more robust strategy for guiding LLM decision-making in these contexts.
Also Read:
- Unpacking AI’s Moral Compass: How Language Models Prioritize Values
- GAMA: Securing AI Multi-Agent Systems with Smart Anonymization and Enhanced Reasoning
Implications and Future Directions
This work demonstrates that pretrained LLMs can not only reason effectively on their own but also provide statistically significant cooperative gains when tasked with assisting another player. This moves LLM capabilities beyond passive language understanding towards active collaboration, laying groundwork for future LLMs to serve as dynamic, real-time partners in complex, goal-driven environments.
The researchers acknowledge several limitations, including the constrained cooperative mechanics of UNO, the exclusive use of fixed rule-based agents as teammates, and resource constraints preventing the evaluation of even larger or proprietary models like GPT-4. Future work aims to address these by equalizing trial counts for all models, exploring more complex cooperative game environments, expanding the list of models, and pairing LLMs with more adaptive agents or even human players to assess varied teammate dynamics.
For more detailed information, you can read the full research paper available here.
