TLDR: This research paper investigates the optimal trade-off between the number of items (N) and responses per item (K) for reliable machine learning evaluation, especially when human disagreement is a factor. It demonstrates that a modest total annotation budget (N x K <= 1000) with K > 10 often yields reliable results. The study emphasizes that the ideal N-K balance is highly dependent on the chosen evaluation metric, with metrics sensitive to full response distributions benefiting most from higher K values. The findings provide a data-driven methodology for ML practitioners to design more effective and budget-conscious evaluations by strategically accounting for human disagreement.
Reproducibility is a cornerstone of scientific validation, and in the rapidly evolving fields of AI and Natural Language Processing (NLP), it is crucial for building trust and advancing knowledge. However, a significant challenge arises because the ‘ground truth’ responses used in machine learning evaluations often come from humans, among whom disagreement is common. Surprisingly, little research has focused on the impact of effectively ignoring this disagreement, which is typically the case.
One major reason for this oversight is the limited budget for collecting human-annotated evaluation data. Obtaining multiple samples from various annotators for each example significantly increases the cost per item. This leads to a critical question: how much data needs to be collected to ensure statistically reliable testing, especially when considering human variability?
The N, K Trade-off
A recent research paper, titled “Forest vs Tree: The(N, K) Trade-off in Reproducible ML Evaluation”, delves into this very problem. It investigates the trade-off between the number of items (N) and the number of responses per item (K) required for reliable machine learning evaluation, given a fixed budget (N × K). The authors, Deepak Pandita, Flip Korn, Chris Welty, and Christopher M. Homan, from Rochester Institute of Technology and Google Research, analyze diverse categorical datasets with multiple annotations per item, as well as simulated distributions, to pinpoint the optimal (N, K) configuration.
The study addresses several key research questions, including the lowest total number of annotations (N × K) needed for repeatable results when comparing two machine learning models, and how this number depends on factors like response distribution, evaluation metrics, number of categories, and statistical instruments (null hypothesis significance tests vs. confidence intervals).
Methodology and Key Findings
The researchers developed a simulation framework that generates outputs for comparing two models (A and B) against a gold standard (G), with responses being categorical rather than continuous. They employed a Dirichlet-categorical distribution, which naturally models the probability of observing categories and simplifies Bayesian inference. This approach allows for robust estimates, especially with limited data.
Their experiments involved varying N × K values (from 100 to 50,000) and K (from 1 to 500) across different metrics and perturbation levels. They used five real-world datasets: Toxicity, DICES, D3code, JobsQ1, and JobsQ3, along with synthetic data.
The findings are insightful:
- Accounting for human disagreement can be achieved with a surprisingly modest budget. An N × K value of no more than 1000 (and often much lower) was found to be sufficient for every dataset tested on at least one metric.
- Crucially, this minimal N × K almost always occurred when K (responses per item) was greater than 10. This suggests that having more human annotations per item is often more beneficial than simply having more items.
- The nature of the trade-off between K and N, and whether one even existed, heavily depended on the evaluation metric used. Metrics that are more sensitive to the full distribution of responses performed better at higher levels of K.
Specifically, for Accuracy, the p-value tends to increase with K, suggesting that a single or very small K is often most reliable. For Total Variation (TV), higher K generally leads to lower, more reliable p-values, with improvements seen even at K=300. Wins metrics showed optimal p-values for K between 25 and 100, while confidence intervals improved with increasing K. KL-Divergence showed complex behavior, with confidence intervals improving with K, but p-values exhibiting initial peaks before settling down.
The study highlights that the TV metric often required the smallest N × K overall, and this was achieved with a K value greater than 10.
Also Read:
- Understanding AI Grading Uncertainty with Semantic Entropy
- AI Judging AI: A New Era for Language Model Evaluation
Implications for ML Practitioners
The methods presented in this paper can significantly help machine learning practitioners design more effective and budget-conscious evaluations. By understanding the optimal metrics and the right balance between the number of items and annotations per item, they can collect test data that yields the most reliability for their budget. The research underscores that moving beyond the single-truth paradigm and strategically collecting multiple responses is not a luxury, but a fundamental aspect of robust and meaningful machine learning evaluation. Embracing human disagreement is key to building greater trust and confidence in model performance.
For more detailed information, you can read the full research paper here.
