TLDR: Rezwan is a large-scale, AI-assisted Hadith corpus comprising over 1.2 million narrations, developed using Large Language Models (LLMs) for automated processing, segmentation, validation, and multi-layer enrichment. The project includes machine translation into 12 languages, intelligent diacritization, abstractive summarization, thematic tagging, and cross-text semantic analysis. Expert evaluation showed near-human accuracy in structured tasks and significant superiority over existing manually curated corpora. The AI-driven approach demonstrates economic feasibility, completing tasks equivalent to over 700,000 person-hours of expert labor within months, thus transforming the accessibility and analysis of Islamic heritage for digital humanities and Islamic studies.
A groundbreaking initiative named Rezwan has emerged, leveraging the power of Large Language Models (LLMs) to create an extensive and richly annotated corpus of Hadith texts. This project addresses the long-standing challenges in processing and analyzing Hadith, which are the sayings and traditions of the Prophet Muhammad (pbuh) and the Imams (as), serving as the second most authoritative source of Islamic knowledge after the Qur’an.
Traditionally, studying Hadith has been a labor-intensive and complex endeavor due to the vast scale, scattered nature of sources, and intricate linguistic and structural characteristics of these texts. Rezwan introduces a fully automated, AI-assisted pipeline that transforms raw Hadith texts into a research-ready infrastructure, making this invaluable Islamic heritage more accessible and analyzable than ever before.
The Automated Pipeline: From Raw Text to Rich Annotation
The core of the Rezwan project is an intelligent, multi-stage pipeline designed to process over 1.2 million narrations. The process begins with data collection from well-known digital repositories like Maktabat Ahl al-Bayt, chosen for its comprehensiveness and structural consistency. Despite these advantages, an initial manual filtering and reclassification step was necessary to ensure all texts containing narrations were correctly identified.
Once collected, the texts enter the automated pipeline, which includes:
- Segmentation and Hadith Boundary Detection: LLMs are employed to accurately separate the ‘chain’ of narrators (isnad) from the ‘main text’ of the narration (matn), a critical step given the variability of Hadith structures.
- Validation and Alignment: Extracted narrations are rigorously validated against original sources using fuzzy string matching to account for minor textual variations or OCR errors, ensuring fidelity.
- Automated Enrichment: This is where Rezwan truly shines, adding multiple layers of analytical and linguistic metadata.
Multi-Layered Enrichment for Deeper Understanding
Each narration in the Rezwan corpus is enhanced with several innovative features:
- Machine Translation: Narrations are translated into 12 major world languages, including English, Persian, Turkish, Urdu, French, Spanish, and German, significantly broadening accessibility for non-Arabic speakers.
- Intelligent Diacritization: An AI-based model applies diacritics to over 80% of texts that originally lacked them, improving readability and linguistic accuracy.
- Summarization and Thematic Tagging: Concise summaries, key points, and thematic labels are generated for each narration, enabling researchers to quickly navigate and filter large datasets.
- Semantic and Lexical Relationship Discovery: Using vector-based semantic similarity, narrations are clustered to uncover hidden lexical, semantic, and thematic connections that go beyond simple keyword searches.
- Quality Control Filters: Automated filters are in place to flag anomalous outputs, ensuring data reliability.
Rigorous Evaluation and Superior Performance
To ensure scientific reliability, Rezwan underwent a rigorous evaluation process. A random sample of 1,213 narrations was assessed by six domain experts using a standardized evaluation form that combined numerical scoring (0–10 scale) with qualitative annotations. The corpus achieved an impressive mean overall score of 8.46 out of 10.
The evaluation highlighted strong performance in structured tasks such as chain–text separation (9.30) and summarization (9.33), demonstrating near-human accuracy. While more interpretive tasks like semantic similarity (7.28) and diacritization (2.49% character-level error rate) presented ongoing challenges, they were deemed acceptable given their complexity.
A comparative analysis against the manually curated Noor Corpus, a respected resource in Hadith studies, further underscored Rezwan’s advantages. Rezwan significantly outperformed the Noor Corpus, which scored 3.66, primarily due to its comprehensive, multi-layered enrichment. While the Noor Corpus was strong in foundational tasks, it lacked most of the analytical layers provided by Rezwan’s AI-driven approach.
Also Read:
- Advancing Kurdish Language Understanding with a New Semantic Textual Similarity Dataset
- AI’s Nuanced View: How Reasoning LLMs Personalize Offensiveness Detection in Political Discourse
Economic Feasibility and Future Implications
Beyond its quality, Rezwan demonstrates immense economic value. The project made possible what would be practically infeasible with traditional manual methods. It is estimated that the AI pipeline produced a corpus whose quality is equivalent to approximately 700,000 person-hours of expert labor, a task that would require a team of 10 experts working full-time for over 70 years. The AI pipeline completed this within months at a fraction of the cost, proving its economic optimality.
The Rezwan Corpus represents a significant leap forward for digital humanities and Islamic studies. It offers a unified, richly annotated dataset that supports comparative studies, thematic analyses, and cross-linguistic research on an unprecedented scale. For AI research, it provides valuable testbeds for improving LLMs in high-stakes textual domains. The methodology showcases how automation can amplify human scholarship, making vast textual traditions computationally accessible while maintaining scholarly rigor.
While challenges in diacritization and semantic similarity persist, future work aims to iteratively fine-tune LLMs, expand multilingual coverage, and adapt the pipeline to other related Islamic corpora. This will pave the way for an ecosystem of intelligent research tools that will meaningfully accelerate the exploration of Islamic heritage. For more details, you can refer to the original research paper: Rezwan: Leveraging Large Language Models for Comprehensive Hadith Text Processing.
