TLDR: This research paper explores pre-hoc prediction in AutoML, a method to intelligently pre-select machine learning models for tabular datasets, thereby reducing the extensive computational demands of traditional post-hoc AutoML. The authors propose a novel framework that leverages both traditional machine learning models and Large Language Model (LLM) agents. By integrating statistical metadata and textual descriptions from OpenML datasets, their approach aims to narrow down the search space for optimal models. Experiments on the AWS AutoGluon portfolio dataset show that traditional methods, particularly those using dataset descriptions like RoBERTa, perform well in predicting model families. LLM agents, while outperforming random baselines, require further refinement, with Llama in a Zero-Shot setting showing promising results. The study demonstrates the value of dataset characterization for efficient AutoML and suggests future directions for enhancing LLM reasoning in this domain.
The field of Automated Machine Learning (AutoML) has seen significant advancements, particularly in selecting the best-performing models for a given dataset. Traditionally, this process, known as post-hoc model selection, involves extensive hyperparameter searches where various models are trained and tested on the target dataset. While effective, these methods demand substantial computational resources, posing a barrier for users with limited access to powerful computing infrastructure.
A promising alternative emerging to address these challenges is pre-hoc prediction. This approach aims to intelligently pre-select models, thereby bypassing the need for exhaustive searches and significantly reducing computational overhead. Despite its potential, pre-hoc prediction has remained relatively underexplored in academic literature.
A recent research paper, Pre-Hoc Predictions in AutoML: Leveraging LLMs to Enhance Model Selection and Benchmarking for Tabular datasets, delves into this under-explored area. Authored by Yannis Belkhiter, Seshu Tirupathi, Giulio Zizzo, Sachin Sharma, and John D. Kelleher, the paper investigates the intersection of AutoML and pre-hoc model selection by integrating traditional models and Large Language Model (LLM) agents. Their goal is to reduce the search space of AutoML libraries by utilizing dataset descriptions and statistical information.
A Novel Framework for Pre-Hoc AutoML
The researchers propose a novel framework that addresses key gaps in pre-hoc AutoML research. Their contributions include the integration of diverse dataset information, combining structured statistical data with textual dataset descriptions, and employing retrieval-augmented generation (RAG) for efficient and accurate pre-hoc model selection. Furthermore, they developed an AutoML agent leveraging LLMs with enhanced context, creating a pipeline for solving tabular AutoML problems. This agent not only predicts suitable models but also provides reasoning for its selections, contributing to explainability in AutoML.
Leveraging Dataset Information
The core idea is to use numerical and textual descriptions of a dataset to suggest a high-performing model to the user. The paper explores two main types of supplementary information:
- Statistical Metadata: This includes statistical variables like the number of samples, features, missing values, class imbalance, and target entropy, which help infer pre-hoc information about model performances.
- Textual Dataset Description: Each dataset, sourced from OpenML, comes with a dataset card containing its name, domain, feature information, task type, and references. This textual data provides rich context for model selection.
Experimental Setup and Strategies
The methodology was applied to the AWS AutoGluon portfolio dataset, a benchmark comprising 175 tabular classification datasets available on OpenML. The experiments compared two primary strategies for pre-hoc model selection:
- Traditional Pre-Hoc Predictors (Pre-HP): These methods involve training conventional machine learning models (like Euclidean Distance, K-Nearest Neighbors, Random Forest Classifier for metadata, and TF-IDF, BERT, RoBERTa encodings, and BERT Classifier for textual descriptions) to predict the most performant model based on the TabRepo ranking.
- LLM Pre-HPs (AutoML Agent): This approach utilizes LLMs, specifically Granite-3.1-8b, Llama-3.1-8b, and GPT-4o, to select models based on dataset metadata and RAG content about available tabular models. The LLMs were tested in Zero-Shot (no examples) and Few-Shot (with one metadata example per model) configurations, both with and without RAG.
The performance of these approaches was evaluated using two metrics: accuracy in selecting the correct model family and accuracy in selecting the exact ground-truth model. These were compared against two baselines: a random pick and selecting the most frequent label in the portfolio.
Also Read:
- Assessing LLM Reliability in Tabular Feature Engineering: A Multi-level Approach
- Enhancing Anomaly Detection with Semantic Context in Tabular Data
Key Findings and Future Directions
The evaluation revealed that most traditional Pre-HP models outperformed the baselines for both family and model selection, indicating the value of statistical information and dataset descriptions. Notably, the RoBERTa model, using dataset descriptions, achieved a strong family accuracy of 0.61.
The LLM-based AutoML agent, while outperforming the random baseline, showed lower performance compared to traditional models. This is partly expected, as traditional models had access to more historical knowledge. Among the LLMs, Llama in a Zero-Shot No RAG setting demonstrated the best overall performance, and Granite showed more consistency across different settings. The addition of RAG had a notable impact on Few-Shot settings for GPT and Llama, increasing Family accuracy. Interestingly, Few-Shot configurations did not consistently outperform Zero-Shot settings.
In conclusion, the research highlights the promising potential of both traditional methods and LLMs for improving pre-hoc model selection in AutoML. The study underscores the importance of dataset characterization through metadata and textual descriptions for inferring model performance. While traditional models currently benefit more from enriched dataset information, LLMs exhibit reasoning capabilities in AutoML problems that, with further refinement, could lead to more efficient and explainable AutoML workflows. This work paves the way for advanced AutoML approaches and emphasizes the value of open-source datasets with consistent textual and numerical information.
