TLDR: Wasm is a novel pipeline that creates high-quality, structured Arabic interleaved multimodal datasets from Common Crawl. It addresses the critical lack of such resources for Arabic by preserving document structure, interleaving text and images, and employing Arabic-tuned filtering and granular deduplication methods. This approach enables more effective training for large language and multimodal models, fostering advancements in Arabic AI.
The advancement of artificial intelligence, particularly large language models (LLMs) and large multimodal models (LMMs), relies heavily on the quality and quantity of their training data. While the internet offers a vast ocean of information, much of it is unstructured and noisy, especially for languages other than English. For Arabic, the challenge is even greater due to a significant lack of high-quality multimodal datasets that preserve the original document structure.
A new research paper introduces “Wasm,” a novel pipeline designed to address this critical gap. Wasm, which means ‘tag’ or ‘mark’ in Arabic, reflects its unique ability to preserve web markup structures. This pipeline processes the massive Common Crawl dataset to create a new Arabic multimodal dataset that provides structured markdown output, a significant departure from existing Arabic corpora that typically focus only on text extraction.
Why Structured Multimodal Data Matters
Traditional Arabic datasets often discard valuable structural cues like document layout, formatting, and image associations. However, recent studies highlight that interleaved image-text data, which maintains the natural sequence of textual and visual elements within documents, is crucial for training advanced multimodal models. This structure allows models to learn long-range dependencies, maintain narrative coherence, and align images with text across various segments, leading to richer multimodal reasoning capabilities.
Wasm is the first Arabic pipeline to produce both interleaved multimodal datasets and text-only corpora with full structural preservation. It retains the document-level structure and the natural interleaving of text and images as they appear on webpages, including layout cues and image-text associations vital for multimodal training. The framework builds upon the OBELICS framework but adapts and extends it specifically for Arabic web data and multimodal use cases. Unlike OBELICS, which outputs plain text, Wasm converts web content into structured Markdown with interleaved images, offering flexibility for both text-only and multimodal pre-training.
How Wasm Works: A Simplified Overview
The Wasm pipeline involves several key stages:
- Metadata Extraction: It starts by filtering web pages containing Arabic content from Common Crawl dumps, extracting metadata like URL, storage location, and language. This early filtering saves significant computational resources.
- HTML Processing and Standardization: Raw webpage content is converted to HTML, then cleaned by normalizing whitespace, removing HTML comments, structural elements (like headers, footers, navigation bars), and CSS content to reduce noise.
- Simplifying and Structuring Web Content: The cleaned content is converted into Markdown format. Crucially, Wasm transforms text into structured text (headers, paragraphs, lists, tables) and categorizes visual elements (figures, images), maintaining semantic coherence by concatenating text elements with identical tags.
- Quality Filtering: A multilevel filtering system is applied at both tag and document levels. Wasm introduces manual modifications to adapt filtering to Arabic, such as relaxing or removing thresholds for word repetition, stopword ratio, punctuation ratio, and common word ratio, which behave differently in Arabic compared to English. A stricter, custom-trained Language Identification process ensures predominantly Arabic text, while a meticulously calibrated Perplexity Threshold eliminates incoherent or machine-generated content.
- Visual Data Filtering: Instead of downloading images, Wasm collects their URLs, focusing on site-level blacklisting for inappropriate content, a conservative approach given the scarcity of Arabic multimodal resources.
- Tag-Level Deduplication: Unlike pipelines that reject entire documents for duplication, Wasm uses the Needleman–Wunsch algorithm to identify and remove nearly duplicate content at the HTML tag level, preserving documents with unique content alongside repeated elements.
Also Read:
- IndicVisionBench: A New Frontier for Evaluating AI’s Cultural and Multilingual Understanding in India
- Automating the Detection of Modality Bias in Multimodal Misinformation
Key Innovations and Impact
Wasm introduces three fundamental improvements over existing methodologies:
- Structured Data Preservation: It preserves the hierarchical structure inherent in web documents in Markdown format, maintaining semantic relationships crucial for models to understand document-level coherence.
- Enhanced Perplexity-Based Quality Assessment: Wasm refines the perplexity-based filtering strategy with an Arabic-tuned KenLM model, trained on a balanced corpus emphasizing linguistic fidelity and diversity across Arabic dialects and topics. This ensures the authenticity of human-produced Arabic text.
- Granular Node-Level Deduplication: By performing deduplication at the HTML node level, Wasm preserves documents containing unique content alongside duplicated elements, significantly improving content diversity and processing efficiency.
By releasing both the dataset and the pipeline code, Wasm democratizes access to advanced multimodal Arabic resources. This initiative not only pushes the boundaries of Arabic NLP development but also enables reproducible research and lays the groundwork for future large-scale corpus construction. For more details, you can read the full research paper here.
