TLDR: A new research paper argues that current benchmarks for Source Independent Transferability Estimation (SITE) metrics are flawed. These benchmarks use unrealistic model spaces, lead to static model rankings that can be beaten by simple heuristics, and fail to assess if score differences meaningfully correlate with performance gaps. The paper proposes best practices for creating more robust benchmarks, including diverse model and dataset spaces, and engineering for varied model performance across tasks, to ensure more realistic and useful evaluation of transferability metrics.
In the rapidly evolving world of artificial intelligence, particularly in deep learning, the practice of using models pre-trained on vast datasets has become commonplace. This approach, known as transfer learning, allows developers to adapt powerful models to new, specific tasks without starting from scratch. A crucial challenge within this domain is selecting the best pre-trained model for a given target task without the time-consuming process of fine-tuning or access to the original source dataset. This is where Source Independent Transferability Estimation (SITE) metrics come into play, offering a way to predict a model’s performance cheaply.
However, a recent research paper titled “How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluation” by Prabhant Singh, Sibylle Hess, and Joaquin Vanschoren, critically examines the very foundations upon which these SITE metrics are evaluated. The authors argue that the widely adopted benchmarks used to measure progress in transferability estimation are fundamentally flawed, leading to an inflated perception of existing metrics’ performance.
The Flaws in Current Benchmarking
The paper identifies three major shortcomings in the standard benchmark setups:
Unrealistic Model Search Space: Current benchmarks often feature model pools dominated by variants of just two architectural families, like ResNets and DenseNets, primarily differing in size. This setup simplifies the model selection problem to merely identifying the largest model, as larger models predictably outperform smaller ones within the same family. Furthermore, the inclusion of models designed for edge computing, such as MobileNet and MNASNet, which consistently rank at the bottom, unnecessarily inflates the search space without adding meaningful challenge. The researchers recommend excluding edge-oriented models and using at most one representative architecture per family, ensuring models have similar sizes for a fairer comparison.
Solved by a Static Ranking: A significant finding is that the standard benchmark can be “solved” by a simple, static ranking heuristic. This means that a few high-capacity models consistently occupy the top ranks across almost all target datasets, regardless of the specific task. The paper demonstrates that a naive static ranker, which orders models based on size and an alternation of ResNet and DenseNet families, significantly outperforms sophisticated SITE metrics. This suggests that the benchmark rewards memorizing a fixed model hierarchy rather than truly assessing a metric’s ability to perform task-specific transferability estimation.
Lack of Fidelity to Accuracy Differences: Beyond just ranking, a practical transferability metric should provide scores whose magnitudes are meaningful. A large difference in metric scores should ideally correspond to a large difference in actual downstream accuracy. However, the standard evaluation protocol, which focuses solely on rank correlation (like Kendall’s Tau), overlooks this crucial property. The paper shows that many existing metrics exhibit a weak correlation between score differences and accuracy differences, making it difficult for users to confidently interpret scores to make informed decisions about model selection.
Also Read:
- Enhancing LLM Training: Focusing on Local Steps for Better Reasoning
- The Unexpected Truth About Prompting LLMs for Consistent Evaluations
Recommendations for Robust Benchmarks
To address these critical issues, the authors propose a set of actionable best practices for constructing more robust and practically relevant benchmarks:
Construct a Diverse and Non-Trivial Model Space: Benchmarks should include models from architecturally distinct paradigms, such as Convolutional Neural Networks (ConvNeXt), Vision Transformers (ViT, Swin), and MLP-based models (MLP-Mixer). Crucially, these models should have comparable computational budgets (e.g., similar parameter counts or FLOPs) to ensure that the evaluation truly isolates the impact of architectural inductive bias rather than just model size.
Ensure a Diverse and Challenging Dataset Space: The suite of target tasks must be varied and challenging. This includes datasets with sufficient “performance headroom” where even strong models don’t achieve near-perfect accuracy, allowing for statistically significant performance gaps to be measured. Datasets should also span diverse visual domains, such as fine-grained classification, medical imaging, satellite imagery, and texture analysis, to test a metric’s generalization capabilities.
Engineer for Performance Spread and Rank Dispersion: A good benchmark should avoid a static model hierarchy where the same models always win. Instead, it should be designed so that model rankings vary substantially across tasks, with different architectures excelling in different contexts. This forces transferability metrics to make nuanced judgments about the best architecture for a given task.
Release Code and Data: For full reproducibility and to reduce inconsistencies, researchers should release the code for their metrics, datasets, obtained scores, final accuracies, and pre-trained models.
This research serves as a vital call to action for the machine learning community to adopt more rigorous standards in benchmarking SITE metrics. By implementing these recommendations, future research can develop transferability metrics that are genuinely useful to practitioners, providing truly predictive and reliable guidance for real-world model selection. You can read the full paper for more details: How NOT to benchmark your SITE metric: Beyond Static Leaderboards and Towards Realistic Evaluation.
