TLDR: A new AI-powered system has been developed to automatically classify page images from historical documents, addressing the challenges of manual sorting and processing in large digital archives. The system, which leverages fine-tuned CLIP models, achieved near-perfect accuracy in categorizing diverse document pages (e.g., text, drawings, tables, photos). This automation significantly reduces manual effort and enables efficient, content-specific downstream analysis for historical document collections.
Digitization efforts in libraries, archives, and museums have led to an explosion of digital images from historical documents. While this improves access, managing and analyzing these vast and diverse collections poses significant challenges. These documents often contain a mix of handwritten, typed, and printed texts, along with various graphical elements like drawings, maps, and photographs, all presented in different layouts. Manual sorting and analysis of such heterogeneous data are incredibly time-consuming and prone to human error, especially given the sheer volume of pages.
The Challenge of Historical Document Processing
The digital archives derived from historical documents come with unique characteristics that complicate their management. Collections span long historical periods, with data volume growing exponentially. Scans are often high-resolution, resulting in large files. However, the original documents’ physical condition, such as aging or damage, can introduce visual defects like low contrast, page skew, bleed-through, water damage, tears, and even stamps or manual annotations that obscure content. Furthermore, metadata is often scarce, with filenames typically encoding technical details rather than semantic content like author or title.
These issues create major hurdles: manual sorting is impractical, reviewing each page is prohibitively time-consuming, and different page types require different analysis tools. For instance, Optical Character Recognition (OCR) works well for clean text but struggles with noisy, handwritten, or complex historical pages. Without an initial classification, applying the correct downstream processing pipelines (like OCR for text, layout analysis for tables, or image analysis for graphics) efficiently is impossible.
An Automated Solution for Page Classification
To address these challenges, researchers developed an automated system for classifying page images from historical archives based on their visual content and layout. The goal was to create a tool that enables more efficient organization, targeted information retrieval, and the application of content-specific analysis techniques for both historical and modern page scans.
Evolving Classification Categories
The project involved a meticulous process of defining and refining classification categories. Initially, categories were broad, distinguishing between figures, tables, and general text. Through iterative feedback from domain experts from the Institute of Archaeology of the Czech Academy of Sciences, these categories were expanded and refined. The final system uses 11 distinct categories, designed to capture relevant variations for specific processing workflows. These include categories for drawings (DRAW, DRAW L), various types of text lines in tables (LINE HW, LINE P, LINE T), photographs (PHOTO, PHOTO L), and different forms of plain text (TEXT, TEXT HW, TEXT P, TEXT T). A priority order was established to handle pages that might fit multiple categories, prioritizing structured or visually distinct content.
From Traditional Methods to Deep Learning Breakthroughs
The development process explored various approaches. A computationally efficient baseline using handcrafted features and a Random Forest classifier achieved about 75% accuracy, which was deemed insufficient for reliable automated sorting. This highlighted the need for more powerful methods.
The team then turned to deep neural network architectures. Modern Convolutional Neural Networks (CNNs) like EfficientNetV2 and RegNetY, along with Transformer-based models such as Vision Transformer (ViT) and Document Image Transformer (DiT), were fine-tuned on the dataset. These models significantly improved performance, consistently achieving accuracies above 97%, with some reaching nearly 98%.
The most remarkable results came from the CLIP (Contrastive Language–Image Pre-training) models. While their out-of-the-box performance in a zero-shot setting was limited (below 50%), fine-tuning these models for just 10 epochs on the annotated dataset led to near-perfect accuracy, with some variants achieving 100% accuracy. This demonstrated CLIP’s exceptional effectiveness when adapted to domain-specific tasks. The CLIP B/16 (224) model was ultimately chosen for deployment due to its optimal balance of accuracy and computational cost.
Also Read:
- Enhancing VLM Performance Through Smart Data Filtration
- AI Agents Collaborate to Understand Long Documents
A Practical System for Archives
The developed system is designed for practical integration into existing archival processes. It is implemented as a Python application using PyTorch and Hugging Face’s Transformers library, available in a public GitHub repository. It offers a modular architecture with configuration management, command-line interfaces, and cross-platform scripts for data preparation, including converting PDFs to images and sorting annotated data.
The system can process single image files or entire directories in batches, optimizing for memory efficiency. It runs on-premises, which is crucial for sensitive collections, and can operate on standard CPU hardware for inference, though a GPU is recommended for faster processing. On a modern office desktop, it can process hundreds of thousands of pages in under a week.
The classification logic was refined in collaboration with archival experts to ensure it meets their practical needs. For example, all significant photographs or drawings must be identified, tables and forms detected regardless of borders, and clean text pages distinguished from those with mixed content or minor annotations. This automated labeling allows for efficient routing of documents to subsequent processing stages, such as OCR for text or specialized image analysis for graphics, significantly reducing manual effort and increasing throughput.
This work provides a robust and accurate solution for automated page-image classification in historical archives, paving the way for significantly reduced manual effort in historical document workflows and supporting digital humanities research and mass digitization initiatives. You can find more details about this research in the paper: Page image classification for content-specific data processing.
