TLDR: OCR-Quality is a new, comprehensive human-annotated dataset designed to evaluate and develop OCR quality assessment methods. It comprises 1,000 diverse PDF pages converted to high-resolution images, processed by a state-of-the-art VLM, and manually scored on a 4-level quality scale. Covering multiple languages and document types, this dataset addresses the critical need for reliable OCR quality assessment in real-world applications, providing a valuable benchmark for training and evaluating OCR verification systems.
Optical Character Recognition (OCR) technology has become a cornerstone in how we process and understand documents, from digitizing old archives to extracting information for various applications. Despite significant advancements, especially with the rise of Vision-Language Models (VLMs), accurately assessing the quality of OCR outputs remains a complex challenge. This is particularly true in real-world scenarios where documents come in many forms, languages, and layouts.
Existing methods for evaluating OCR often focus on average accuracy across standardized tests, which doesn’t always provide a clear picture of how reliable individual OCR predictions are. This gap can lead to errors propagating through downstream applications, reducing overall system reliability. To address this critical need, a new dataset called OCR-Quality has been introduced. This comprehensive, human-annotated dataset is specifically designed to help researchers develop and evaluate better OCR quality assessment methods.
What is OCR-Quality?
OCR-Quality is a meticulously curated dataset consisting of 1,000 diverse PDF pages. These pages were converted into high-resolution PNG images (300 DPI) to preserve visual detail. The documents were sampled from a wide array of real-world sources, including academic papers, textbooks, e-books, and multilingual content, ensuring a broad representation of document characteristics.
Each document in the dataset was processed using Qwen2.5-VL-72B, a cutting-edge Vision-Language Model, with a specialized OCR prompt designed to maintain document structure, handle multi-column layouts, and support mathematical notation. Following this, each OCR output was manually annotated by humans, assigning a quality score using a clear 4-level system:
- Score 1 (Excellent): Near-perfect OCR with minimal or no errors.
- Score 2 (Good): Minor errors that do not affect understanding.
- Score 3 (Fair): Some noticeable errors, but the content is still usable.
- Score 4 (Poor): Significant errors that severely affect content quality.
This human annotation is crucial as it provides a ground truth for evaluating how well automated systems can judge OCR quality.
Diversity and Characteristics of the Dataset
The dataset boasts impressive diversity, covering:
- Languages: Chinese (educational materials, textbooks, e-books), English (academic papers, textbooks, literature), and Multilingual documents.
- Document Types: Academic papers with complex layouts, textbooks with mixed text and equations, e-books with varied formatting, and educational materials with diagrams and tables.
- Formatting Complexity: Includes single and multi-column layouts, mathematical expressions, tables, figures with captions, mixed language content, and various font sizes and styles.
The distribution of quality scores within the dataset is also well-balanced, with 50.7% rated as Excellent, 30.5% as Good, 8.4% as Fair, and 10.4% as Poor. This ensures that the dataset provides ample examples of both high-quality and challenging OCR cases, which is vital for robust evaluation.
How Can OCR-Quality Be Used?
Researchers can leverage OCR-Quality for several key evaluation tasks:
- Correlation Analysis: To measure how well predicted quality scores align with human scores.
- Classification Performance: To evaluate how accurately models can classify OCR outputs into acceptable or unacceptable quality categories.
- Ranking Evaluation: To assess a model’s ability to rank documents by their OCR quality.
The dataset also supports various research and practical applications. In research, it can be used to develop new quality assessment methods, quantify uncertainty in OCR outputs, select the best OCR models for specific tasks, and analyze common failure modes. Practically, it can help filter low-quality OCR outputs in document processing pipelines, implement automated quality control, prioritize documents for human review, and identify challenging cases for model improvement.
Also Read:
- Unpacking T2I-RiskyPrompt: A New Benchmark for Text-to-Image Model Safety
- New Benchmark Reveals Vision-Language Models Struggle with Subtle Object State Recognition
Accessing the Dataset
The OCR-Quality dataset is publicly available for research purposes. It can be downloaded from HuggingFace, and more details can be found in the research paper itself. The dataset is provided in Parquet format, including embedded images, OCR text, human scores, and detailed metadata. You can read the full research paper here: OCR-Quality: A Human-Annotated Dataset for OCR Quality Assessment.
While the dataset currently has some limitations, such as its size (1,000 samples) and reliance on a single VLM for OCR processing, the creators plan to expand it significantly in the future. This includes increasing the number of samples, incorporating outputs from multiple OCR systems, adding multi-annotator scores, and extending language coverage. OCR-Quality represents a significant step forward in building more reliable and trustworthy OCR systems for real-world applications by providing a much-needed benchmark for quality assessment.
