Unlocking Ancient Chinese: A New Digital Character List and Corpus for Language Models

TLDR: A new research paper introduces InteChar, a unified and extensible character list that integrates unencoded oracle bone characters with traditional and modern Chinese. This, along with the OracleCS corpus, provides a foundation for training effective language models on historical Chinese texts. Experiments show that models using InteChar and OracleCS significantly outperform existing methods in understanding and processing ancient Chinese, especially for rare and previously undigitized characters.

Understanding ancient languages is a monumental task, crucial for preserving cultural heritage and deciphering lost histories. However, researchers face significant hurdles, particularly with ancient Chinese texts like Oracle Bone Inscriptions (OBI). These challenges include a scarcity of historical language samples, making it difficult to train effective language models, and the absence of comprehensive character encoding schemes for many ancient scripts. This means that many ancient characters exist only as images, not as machine-readable text.

A new research paper introduces a groundbreaking solution called InteChar, a unified and extensible character list designed to integrate unencoded oracle bone characters with traditional and modern Chinese. This innovation, coupled with a new corpus called OracleCS, aims to revolutionize how we digitize and computationally process ancient Chinese texts.

The Challenge of Ancient Chinese

Traditional methods for studying ancient scripts are often limited by human cognitive and learning capacities, which slow down data processing and character decipherment. While modern natural language processing (NLP) has shown promise, it struggles with ancient texts due to several factors:

• Many OBI characters are unencoded and stored as images (handwritten or rubbings), not digital text.
• There’s a severe scarcity of ancient language samples; for instance, only about 5,000 complete oracle bone pieces have been found, yielding a mere 15,000 sentences with more than five characters.
• Existing character encoding schemes often cover only high-frequency characters, ignoring rare but semantically significant ones, leading to incomplete representations and information loss.

InteChar: Bridging the Encoding Gap

InteChar is a Unicode-compatible character list that addresses the problem of unencoded ancient characters. Its construction follows a meticulous four-stage process:

1. Initialization: Starts with the official Unicode character set for modern compatibility.
2. Integration of Existing Characters: Incorporates encoded ancient characters from established libraries like the Zhongjian Library, filtering for relevance to the curated corpus.
3. Construction of New Characters: For characters not found in existing standards, a semi-automated pipeline is used. This involves collecting and preprocessing oracle bone images, applying a radical recognition method (identifying subcomponents of characters), expert verification by paleographers, vectorization into scalable graphics, and assigning new internal code points. This radical-based approach significantly speeds up the creation of new character entries.
4. Expert-Guided Proofreading: Domain experts validate InteChar, using tools like Siamese networks to identify and resolve potential duplicates.

The resulting InteChar list contains 11,288 characters, including both standardized and newly encoded ones, providing a robust foundation for digital processing of ancient texts. You can read more about this work in the full research paper: InteChar: A Unified Oracle Bone Character List for Ancient Chinese Language Modeling.

OracleCS: A Rich Corpus for Training

To support the training of historical language models, the researchers also constructed the Oracle Corpus Set (OracleCS). This linguistically curated corpus focuses on oracle bone inscriptions and other ancient Chinese texts. Its creation involves:

• Expert Curation: Paleography and historical Chinese linguistics experts manually select and annotate high-quality samples from archaeological literature and oracle bone rubbings.
• Rich Annotations: Includes glyph-level and semantic annotations, such as radical decompositions and definitions from classical lexicons.
• Data Augmentation: To combat data scarcity, LLM-assisted data augmentation is used to generate instruction-tuning samples. These samples cover various tasks like sentence translation, synonym substitution, and character decomposition, enhancing the corpus’s scale and diversity.

OracleCS comprises approximately 11,288 unique Chinese characters and 173,459 annotated samples, making it a foundational resource for adapting modern language models to low-resource historical scripts.

Also Read:

• Enhancing Language Models with New Vocabulary for Specialized Tasks
• Enhancing Time Series Forecasting with Deep LLM Integration: Introducing Multi-layer Steerable Embedding Fusion

Demonstrated Effectiveness

Extensive experiments were conducted to evaluate InteChar and OracleCS using ten different language models, including BERT, Llama-3-8B, GPT-2, and specialized Chinese LLMs like Qwen-7B-Chat and GLM-4-9B. The evaluation covered two main areas:

• Embedding Evaluation: Tasks like Cloze Completion on Oracle Bone Inscriptions and Commentary-to-Text Retrieval on Canonical Texts assessed how well character-level and sentence-level embeddings captured contextual and semantic information. Models equipped with InteChar consistently showed substantial improvements across all metrics, demonstrating enhanced representation learning, especially for rare ancient scripts.
• Fine-tuning Evaluation: Downstream tasks such as Ancient Chinese Translation, Polysemous Word Matching, and Word Parsing further validated the approach. Models fine-tuned with InteChar consistently outperformed those using original character lists, achieving better generalization and semantic alignment across different linguistic granularities.

These results confirm that InteChar significantly improves language modeling for ancient texts, particularly in handling rare or previously unencoded characters. This work establishes a solid foundation for future research in ancient Chinese NLP, enabling more robust training and analysis of early scripts.