TLDR: SCORE is a novel evaluation framework for generative document parsing systems that overcomes the limitations of traditional metrics. It is interpretation-agnostic, meaning it accurately assesses models that produce semantically correct but structurally diverse outputs. SCORE integrates adjusted edit distance, token-level diagnostics, semantic table evaluation with spatial tolerance, and hierarchy-aware consistency checks to provide a more comprehensive and fair assessment of modern AI document processing capabilities, revealing insights missed by conventional methods.
The world of artificial intelligence is rapidly advancing, particularly in how machines understand and process documents. Traditional document parsing systems, which extract information from documents, have long relied on metrics like Character Error Rate (CER), Word Error Rate (WER), and Tree Edit Distance-based Similarity (TEDS) to measure their accuracy. However, with the rise of multi-modal generative document parsing systems, such as those powered by advanced Vision Language Models (VLMs) like GPT-5 Mini and Gemini 2.5 Flash, these traditional evaluation methods are proving to be insufficient.
The core challenge is that generative models often produce outputs that are semantically correct—meaning they convey the right information—but structurally different from a predefined ‘ground truth’. For instance, a table with merged cells might be represented as a flattened sequence of tokens by one system and as hierarchical HTML markup by another. Both are valid interpretations, but traditional metrics would penalize the structural deviation as an error, leading to a distorted view of the system’s true performance.
Introducing SCORE: A New Approach to Evaluation
To address these limitations, researchers Renyu Li, Antonio Jimeno Yepes, Yao You, Kamil Pluciński, Maximilian Operlejn, and Crag Wolfe from Unstructured Technologies have introduced SCORE (Structural and COntent Robust Evaluation). This innovative framework is designed to be interpretation-agnostic, meaning it embraces the natural diversity in how generative models represent document content while still ensuring semantic accuracy. You can read the full research paper here.
SCORE advances document parsing evaluation in four key areas:
-
Adjusted Edit Distance: This feature provides a more robust evaluation of content fidelity. It tolerates structural reorganizations, recognizing when semantically equivalent content is organized differently, which traditional edit distance metrics would incorrectly penalize.
-
Token-Level Diagnostics: SCORE offers detailed diagnostics at the token level to distinguish between content omissions (missing information) and hallucinations (adding spurious information). This helps in understanding specific error types.
-
Table Evaluation with Spatial Tolerance and Semantic Alignment: For tables, SCORE moves beyond simple spatial overlap. It normalizes diverse output formats (HTML, JSON, structured text) into a common semantic representation, allowing for comparison based on content similarity and positional correspondence, even with minor spatial shifts.
-
Hierarchy-Aware Consistency Checks: This dimension assesses how well systems capture the hierarchical organization of content. It maps heterogeneous element labels into functional categories (e.g., ‘title’, ‘subtitle’, ‘sub-heading’ all map to ‘TITLE’), enabling semantic-level comparison across different labeling schemes.
Experimental Insights
The researchers tested SCORE across 1,114 pages from two datasets: a ‘Mini Holistic’ collection and an ‘Industry Documents’ dataset. They evaluated various document parsing solutions, including VLM-based systems like Gemini 2.5 Flash, GPT-5 Mini, and Claude Sonnet 3.7/4, as well as traditional object detection (OD) based pipelines.
The results consistently showed that SCORE revealed performance patterns and corrected rankings that standard metrics missed. For example, in 2–5% of pages with ambiguous table structures, traditional metrics penalized systems by 12–25% on average. SCORE corrected these cases, showing equivalence between alternative but valid interpretations. It also demonstrated that generative parsing alone can achieve comprehensive evaluation without needing complex object-detection pipelines.
Also Read:
- Z-Scores: A Diagnostic Tool for Improving Disfluency Removal in AI
- Unpacking SPEED: A New Approach to Evaluating Large Language Models
Key Advantages of SCORE
The framework offers several significant advantages:
-
Multi-Dimensional Performance Characterization: It provides a holistic view of system performance, highlighting distinct strengths in content fidelity, spatial reasoning, and structural hierarchy. For instance, Gemini 2.5 showed high adjusted NED and token coverage, while GPT-5 Mini had the lowest hallucination rates.
-
Ranking Corrections: SCORE can re-rank systems more accurately. In one example, GPT-5 Mini, which appeared weaker than Gemini 2.5 Flash by unadjusted NED, actually achieved a higher adjusted NED, revealing its superior semantic accuracy that was previously masked by penalties for interpretive diversity.
-
Interpretation Tolerance Validation: The difference between standard and adjusted NED scores clearly showed SCORE’s ability to differentiate between genuine errors and acceptable interpretive variations.
-
Element Alignment as a Reality-Grounded Metric: OD-based models performed surprisingly well on alignment for large, well-defined elements, narrowing the gap with VLM systems, though their consistency levels were lower for ambiguous structures.
-
Consistency Challenges in Complex Layouts: SCORE acknowledges that high consistency scores are inherently difficult to achieve in complex documents due to multiple plausible interpretations, reinforcing the need for evaluation that tolerates diversity.
In conclusion, SCORE establishes a principled foundation for fair, semantically grounded, and practical benchmarking of modern document parsing systems. By recognizing that divergent outputs do not always imply model failure, it aligns evaluation more closely with human judgment, especially for complex documents where multiple valid interpretations exist.
