TLDR: TAM Bench is a new benchmark for evaluating LLM-based agents on end-to-end machine learning tasks. It features automated task collection from competition platforms, a leaderboard-driven system for objective difficulty modeling, and a multi-dimensional evaluation framework that assesses performance, format compliance, constraint adherence, and generalization. This aims to provide a more realistic and comprehensive assessment of AI agents’ capabilities in ML, addressing limitations of existing benchmarks.
The rapid advancements in large language models (LLMs) have paved the way for general-purpose AI agents capable of automating complex, end-to-end machine learning (ML) workflows. These agents can handle everything from data analysis and feature engineering to model training and competition solving. However, the tools we use to measure their capabilities, known as benchmarks, haven’t kept pace. Existing benchmarks often fall short in terms of task coverage, diversity of domains, how they model difficulty, and the rigor of their evaluation methods. This means they don’t fully capture what these advanced agents can do in real-world scenarios.
To address these limitations, researchers have introduced TAM Bench, a new benchmark designed to be diverse, realistic, and structured for evaluating LLM-based agents on comprehensive ML tasks. This innovative system aims to provide a more accurate and holistic assessment of AI agents’ performance.
Automated Task Collection
One of TAM Bench’s core innovations is its automated task acquisition system. Unlike traditional benchmarks that rely heavily on manual data collection, which is costly and inefficient, TAM Bench uses browser automation and LLMs to automatically gather and structure ML challenges. It pulls tasks from popular platforms like Kaggle, AIcrowd, and Biendata. This system can handle a wide array of task types and data modalities, including tabular data, text, images, graphs, and audio. By automating this process, TAM Bench significantly reduces manual overhead and ensures a scalable way to acquire tasks across various domains.
Leaderboard-Based Difficulty Modeling
Fairly assessing AI agents requires a clear understanding of task difficulty. TAM Bench introduces a novel leaderboard-driven mechanism to model task complexity objectively. It estimates difficulty using real competition data, specifically looking at the number of participants and the dispersion of scores on leaderboards. This approach moves away from subjective, expert-annotated difficulty levels, offering a scalable and objective way to calibrate tasks. For instance, tasks are categorized as Easy, Medium, or Hard based on a calculated difficulty score that considers the average and best scores achieved by participants, as well as the task’s popularity.
Multi-Dimensional Evaluation Framework
To provide a truly comprehensive assessment, TAM Bench employs a multi-dimensional evaluation framework. It goes beyond single performance metrics, which can sometimes lead to ‘reward hacking’ where agents optimize for a score while ignoring other crucial aspects. TAM Bench evaluates agents across four key dimensions:
- Performance: Measured by a weighted average rank, reflecting the agent’s percentile position on competition leaderboards, with weighting to account for task modality imbalance.
- Format Compliance: Assesses whether the agent successfully generates submission files and if those files match the required structure and format.
- Constraint Adherence: Uses an ‘LLM-as-a-Judge’ method to determine if the agent’s final submitted code adheres to specific modeling constraints or ‘special instructions’ provided for each task.
- Task Generalization: Implicitly evaluated through performance across diverse tasks, modalities, and difficulty levels.
This holistic approach ensures that agents are not only high-scoring but also produce valid, compliant, and robust solutions, reflecting real-world requirements.
TAM Bench is available in three subsets: Lite (18 tasks), Medium (54 tasks), and Full (150 tasks). The Lite version is designed for daily benchmarking and comparative studies, offering balanced coverage across modalities and difficulty levels, while the Full version provides extensive coverage for comprehensive evaluations. You can find more details about this research paper here: Towards Adaptive ML Benchmarks.
Also Read:
- Unpacking Human Agency: A New Benchmark Reveals AI’s Impact on Our Control
- AgentGym-RL: A New Framework for Training LLM Agents in Complex Environments
Experimental Insights
The researchers evaluated two open-source AutoML agent frameworks, AIDE and OpenHands, each paired with GPT-4.1 (a closed-source model) and DeepSeek-V3 (an open-source model), using the Lite version of TAM Bench. The experiments revealed several key findings:
- GPT-4.1-based agents generally demonstrated higher reliability and better rates of generating valid submissions, indicating superior adherence to formatting and output conventions.
- DeepSeek-V3 agents, while capable of producing competitive solutions when successful, showed lower submission and validity rates, suggesting less robust code generation.
- Performance across all agents tended to decline significantly as task difficulty increased, particularly for Medium and Hard tasks, highlighting persistent challenges in handling complex problems.
- Agents performed better on conventional data types like Tabular and Text tasks, but showed weaknesses in perceptual modalities such as Image and Audio, and struggled significantly with Graph and Multi-Modal tasks.
- Notably, the OpenHands + DeepSeek-V3 combination achieved a breakthrough on a challenging tabular task called ‘stanford-covid-vaccine’, outperforming the best human team. This suggests that specific agent-model combinations can excel when aligned with task characteristics.
While the initial evaluation was limited to the Lite version, future work plans to expand experiments to the Medium and Full versions, incorporate more recent AutoML systems, and conduct more statistically robust comparisons with averaged results and confidence intervals. TAM Bench represents a significant step towards establishing a key standard for assessing the data science capabilities of LLM-powered agents in complex, realistic ML workflows.
