TLDR: ChessQA is a new, comprehensive benchmark designed to evaluate how well large language models (LLMs) truly understand chess, moving beyond simple move quality. It assesses five key areas: basic rules, tactical patterns, short calculations, positional evaluation, and semantic descriptions. Initial evaluations of current LLMs reveal persistent weaknesses across all categories, especially in complex tactical and positional judgment, though explicit reasoning significantly improves performance. The benchmark aims to provide a consistent framework for diagnosing and improving LLM chess abilities.
Chess has long been a battleground for artificial intelligence, serving as a crucial test for machine intelligence even before the term ‘AI’ was coined. From early programs like Deep Blue to today’s advanced large language models (LLMs), the game’s well-defined rules and objective outcomes make it an ideal environment to gauge an AI’s reasoning, modeling, and abstraction capabilities. However, current methods for evaluating LLMs in chess are often narrow, focusing only on move quality or specific concepts, which makes it difficult to truly understand an AI’s overall chess comprehension.
To address this gap, researchers Qianfeng Wen, Zhenwei Tang, and Ashton Anderson from the University of Toronto have introduced ChessQA, a new comprehensive benchmark designed to assess LLM chess understanding across a wide spectrum of abilities. This innovative benchmark goes beyond simple move evaluations to offer a controlled and consistent setting for diagnosing and comparing how LLMs grasp the intricacies of chess.
What is ChessQA?
ChessQA is structured around five increasingly complex task categories, mirroring the journey a human player takes to master chess. These categories are:
- Structural: This foundational category tests an LLM’s understanding of basic chess rules, such as recognizing legal moves, identifying checks, locating pieces, and updating the board state after moves.
- Motifs: Moving beyond basic rules, this section evaluates an AI’s ability to recognize common tactical patterns like pins, skewers, forks, batteries, and discovered or double checks.
- Short Tactics: Here, LLMs are challenged to solve tactical puzzles from real games, requiring them to find the best move in positions that demand short, targeted calculations.
- Position Judgment: This category assesses an LLM’s long-term strategic reasoning and multi-step planning by asking it to evaluate an entire position and predict a chess engine’s centipawn advantage.
- Semantic: Representing the highest level of abstraction, this task evaluates an LLM’s ability to understand and articulate high-level chess concepts, connecting concrete moves and positions to strategic and tactical ideas through natural language commentary.
The benchmark is designed with comprehensive coverage, spanning various levels of abstraction, and features calibrated difficulty. It uses standard chess notations (FEN, PGN, UCI) and draws data from extensive sources like Lichess Puzzles and Evaluations, as well as expert commentary from ChessBase 17.
Initial Findings: Persistent Weaknesses and the Power of Reasoning
The researchers evaluated 15 contemporary LLMs, including models from Anthropic, DeepSeek, Google, Meta, Mistral AI, OpenAI, and Qwen. The results revealed that most models still exhibit relatively low performance on ChessQA, with only a few achieving over 50% accuracy. GPT-5* (with ‘thinking’ mode enabled) emerged as the top performer, reaching 79.3% accuracy. Interestingly, some state-of-the-art open-source models also ranked highly, demonstrating strong capabilities.
A key takeaway was the significant difference in performance across categories. While the best models could achieve high scores in basic Structural tasks (up to 97%), they struggled considerably with Short Tactics (averaging 17.4%) and Position Judgment. These categories, which demand implicit search and long-term planning, proved to be the most challenging for LLMs.
Crucially, enabling explicit reasoning (or ‘thinking’ mode) consistently improved model performance, yielding an average accuracy gain of 14.7 percentage points. This suggests that LLMs can effectively leverage additional processing to enhance their chess understanding, though the token efficiency of these reasoning processes remains a point for future improvement.
Understanding AI’s Mistakes
An in-depth error analysis of GPT-5* highlighted common failure modes:
- Board-state hallucination: Models sometimes misread the board, confusing piece positions or even inventing pieces.
- Legality reasoning errors: Mistakes in applying basic move legality rules in tactical problems.
- Sound analysis, wrong answer: The model might show good intermediate reasoning but still select a suboptimal final move.
- False ‘no answer’ assertions: Incorrectly concluding that no valid solution exists for a given problem.
The study also found that explicitly providing piece arrangement context significantly improved overall performance, underscoring the severity of the board-state hallucination problem in LLM reasoning.
Also Read:
- QUARCH: A New Benchmark to Evaluate LLM Reasoning in Computer Architecture
- Unpacking LLM Long-Context Abilities: Insights from the LooGLE v2 Benchmark
The Future of AI in Chess
ChessQA provides a valuable diagnostic tool for understanding and improving LLM capabilities in chess. By offering a comprehensive and dynamic benchmark, it allows researchers to pinpoint specific areas of weakness and track progress as models evolve. As AI continues to advance, benchmarks like ChessQA will be essential for ensuring that our machines not only play the game but truly understand it. You can read the full research paper here.
