TLDR: The paper introduces FlexBench, a new framework that redefines AI system benchmarking as a continuous learning task. It extends MLPerf LLM inference, integrates with Hugging Face, and collects results into an Open MLPerf Dataset. This dataset, combined with the FlexBoard visualization tool, enables predictive modeling to help users select optimal and cost-effective software/hardware configurations for AI deployments, addressing the limitations of traditional, static benchmarks in a rapidly evolving AI landscape.
The world of Artificial Intelligence is advancing at an unprecedented pace, bringing with it a constant stream of new models, datasets, and hardware. This rapid evolution presents a significant challenge for traditional benchmarking methods like MLPerf, which often struggle to keep up, making it difficult for organizations to make informed decisions about deploying, optimizing, and co-designing AI systems.
A new research paper, “Framing AI System Benchmarking as a Learning Task: FlexBench and the Open MLPerf Dataset”, proposes a novel approach: treating benchmarking itself as an AI learning task. This perspective suggests that AI models should be continuously evaluated and optimized across diverse environments, considering key metrics such as accuracy, latency, throughput, energy consumption, and cost.
Introducing FlexBench: A Dynamic Benchmarking Framework
To support this vision, the authors Grigori Fursin and Daniel Altunay introduce FlexBench, a modular and open-source extension of the MLPerf LLM inference benchmark. Integrated with Hugging Face, FlexBench is designed to provide relevant and actionable insights. Unlike static benchmarks, FlexBench aims to evolve continuously alongside the AI ecosystem.
FlexBench operates with a unified command-line interface (CLI) and codebase, allowing users to benchmark a wide variety of models and datasets by simply adjusting input parameters. It leverages the MLCommons CMX workflow automation framework, which includes MLPerf automations, to streamline the process of installing dependencies, selecting software and hardware configurations, and continuously observing system behavior.
The Open MLPerf Dataset and FlexBoard for Predictive Insights
A core component of this new approach is the Open MLPerf Dataset. This dataset aggregates benchmarking results and metadata generated by both FlexBench and standard MLPerf. It is openly shared on platforms like GitHub and Hugging Face, enabling collaborative curation, extension, and analysis. This rich dataset can then be used for predictive modeling and feature engineering, allowing practitioners to anticipate how different AI systems will perform under various conditions.
Complementing the dataset is FlexBoard, a visualization tool implemented as a Gradio application. FlexBoard loads the Open MLPerf Dataset and provides powerful predictive modeling and visualization capabilities. This allows users to compare and predict the most efficient and cost-effective software/hardware configurations for different AI models, tailored to their specific requirements and constraints.
Technical Underpinnings and Validation
Technically, FlexBench employs a client-server design, connecting to a vLLM server and building upon MLPerf LoadGen, the official harness for measuring inference performance. It abstracts models and datasets as interchangeable modules, maintaining MLPerf’s rigor while offering greater flexibility. The framework supports standard inference modes (Server and Offline) and reports detailed LoadGen metrics, including throughput, latency distributions, and time-to-first-token (TTFT), all compatible with MLPerf standards. Crucially, FlexBench also provides additional metrics like accuracy, which are vital for further model optimization.
The FlexBench concept has been successfully validated through MLPerf Inference 5.0 submissions. This included benchmarking non-MLPerf LLMs such as DeepSeek R1 Distill LLaMA 8B and LLaMA 3.3 on the OpenOrca dataset, using commodity servers equipped with NVIDIA H100 GPUs. The automation framework demonstrated its ability to rapidly switch between models, datasets, and hardware configurations without requiring code modifications.
Also Read:
- Optimizing LLM Performance: Balancing Speed and Cost with Dynamic Compute Allocation
- Evaluating AI Agents: Introducing MCP-AgentBench for Real-World Tool Use
Looking Ahead
FlexBench, FlexBoard, and CMX are still in their early stages, with plans to expand support for more models, datasets, and system configurations. Future work also includes enriching the Open MLPerf dataset with more features like model graphs and compiler optimizations to improve predictions. The ultimate goal is to empower anyone to run AI models efficiently and cost-effectively, aligning with their available resources and constraints, and to assist hardware manufacturers in co-designing more energy-efficient AI systems.
