TLDR: A study on Weisfeiler-Leman Features (WLFs) for AI planning analyzed 1,000,000 planning runs to find optimal hyperparameters. It identified a robust set of WLF settings that prioritize minimizing execution time and model size for best planning performance, rather than maximizing expressivity. The research also found no strong correlation between training metrics and actual planning performance.
In the evolving landscape of artificial intelligence, the ability for autonomous systems to learn and plan effectively is crucial. A recent study delves into a classical machine learning technique known as Weisfeiler-Leman Features (WLFs), which has shown promise in guiding planning and search algorithms. This research, conducted by Dillon Z. Chen, explores the intricate details of WLF hyperparameters through an extensive study involving a million planning runs.
WLFs are a method for extracting meaningful information from graph representations of planning tasks. Essentially, a planning problem is first converted into a graph, and then the Weisfeiler-Leman (WL) algorithm processes this graph to generate feature vectors. These feature vectors are then used to learn heuristic functions, which are crucial for efficient search in planning. The advantage of WLFs lies in their speed and expressive power, often outperforming deep learning alternatives in symbolic planning.
The paper introduces and thoroughly investigates various hyperparameters that influence WLF performance. These are categorized into internal and external factors. Internal hyperparameters relate directly to the WL algorithm itself, such as the specific WL algorithm variant used (e.g., vanilla WL, iWL, 2-LWL), the number of iterations the algorithm performs, techniques for pruning features, and the type of hash function employed. External hyperparameters, on the other hand, concern the broader learning-to-plan pipeline, including how the state information is represented (partial or complete) and the optimization method used to train the prediction model (e.g., Lasso, GPR, SVR, or ranking formulations like rkSVM).
To understand the impact of these settings, the researchers conducted a massive experimental analysis, performing over 1,000,000 planning runs across ten different planning domains. This scale allowed for a rigorous empirical understanding of how different hyperparameter combinations affect training, planning performance, and the relationship between training and planning metrics.
Key findings from this extensive study reveal that there is a robust set of hyperparameters that consistently deliver strong performance. Interestingly, the best WLF hyperparameters for learning heuristic functions prioritize minimizing execution time and model size over maximizing model expressivity. Specifically, the most efficient configuration for model size and training time was found to be the vanilla WL algorithm, with 1 iteration, using i-mf feature pruning, a set hash function, partial state representation, and the Lasso optimizer. For overall planning performance, the best combination was identified as the vanilla WL algorithm, 1 iteration, i-mf feature pruning, a set hash function, partial state representation, and the rkSVM optimizer.
Another significant observation was the lack of a strong, statistically significant correlation between training metrics (like evaluation function score, training time, or model size) and actual planning performance. This suggests that good performance during training does not necessarily guarantee superior performance in real-world planning scenarios, a common challenge in classical machine learning methods due to the bias-variance tradeoff.
Also Read:
- Smart Search: How Symmetry-Invariant Novelty Heuristics Boost AI Planning
- Dense vs. Sparse Attention: Navigating Scalability in Graph Transformers
This research provides valuable insights for anyone working with Weisfeiler-Leman Features in planning, offering a guide to selecting effective hyperparameters to achieve optimal results. For more in-depth technical details, you can refer to the full research paper: Weisfeiler-Leman Features for Planning: A 1,000,000 Sample Size Hyperparameter Study.
