TLDR: MultimodalHugs is a new framework built on Hugging Face designed to overcome challenges in Sign Language Processing (SLP) by enabling diverse data modalities and tasks. It addresses issues of reproducibility and complex ad-hoc code in SLP research through standardized dataset formats (TSV), modular processors, and streamlined training/evaluation pipelines. The framework was demonstrated with sign language translation using pose and video inputs, and a novel pixel-based machine translation for Hebrew, showing improved efficiency and flexibility for multimodal tasks.
Sign Language Processing (SLP) is a vital field that aims to develop technologies for the millions of deaf individuals worldwide. However, research in SLP has often been hampered by complex, custom-built code, leading to difficulties in reproducing experiments and making fair comparisons between different approaches. Existing powerful tools like Hugging Face, while excellent for spoken language processing, haven’t been flexible enough to easily integrate the diverse data types found in sign languages, such as video, pose estimations, and other visual features.
Introducing MultimodalHugs
To tackle these significant challenges, researchers have introduced MultimodalHugs, a new framework built on top of the popular Hugging Face ecosystem. This innovative framework is designed to enable more diverse data modalities and tasks, specifically focusing on sign languages, while retaining all the well-known benefits of Hugging Face, such as fast and reproducible experimentation. Beyond SLP, MultimodalHugs also offers a layer of abstraction that makes it applicable to other unique use cases that don’t fit standard Hugging Face templates.
The Need for a Unified Framework
Current SLP research often relies on isolated, task-specific code pipelines. This results in duplicated effort, a feeling of starting from scratch for every new experiment, and a lack of proper documentation. A survey conducted among SLP researchers confirmed these issues, with many reporting that reproducing experiments took over a week, and some never succeeded. Key problems included undocumented preprocessing steps, unavailable open-source code, and framework incompatibilities.
Traditional NLP frameworks, including Hugging Face Transformers, are primarily designed for text-based inputs. Integrating non-textual modalities like pose sequences (skeletal keypoints representing body movements) or raw video frames into these systems requires extensive, custom modifications for preprocessing, data handling, and model input. This often leads to non-reusable and experiment-specific code, hindering progress.
How MultimodalHugs Works
MultimodalHugs streamlines the entire experimentation process through a modular design, ensuring reproducible workflows across various data modalities. It maintains compatibility with Hugging Face’s Trainer, tokenizers, and inference APIs.
- Dataset Standardization: Instead of custom scripts for each dataset, MultimodalHugs uses a unified TSV (Tab-Separated Values) format. Each row in a TSV file defines an example, including fields for the primary input (
signal), optional temporal markers (signal_start,signal_end), text prompts (encoder_prompt,decoder_prompt), and the target output (output). This simplifies integrating multimodal data and allows fine-grained control over samples. - Modular Setup: The framework initializes core components called ‘Training Actors’—Datasets, Processors, and Models—based on a user-defined YAML configuration file. Processors are particularly noteworthy; unlike Hugging Face’s often coupled tokenizers, MultimodalHugs’ processors are modular and model-agnostic, handling modality-specific operations like feature extraction and sequence alignment. Users can define custom models or use default architectures with customizable components like a Feature Extractor, Multimodal Mapper, and Backbone.
- Streamlined Training and Evaluation: Training leverages Hugging Face’s Trainer API, supporting advanced features like mixed-precision and distributed learning. A simple command-line interface instantiates components and runs the training loop. Evaluation is also straightforward, using a
generatecommand to run inference and compute standard metrics (e.g., BLEU, COMET) via Hugging Face’sevaluatelibrary.
Real-World Demonstrations
The researchers showcased the framework’s ease of use and flexibility through two multimodal translation tasks:
- Sign Language Translation: MultimodalHugs was used to replicate and extend a state-of-the-art pose-to-text translation system. By simply organizing the dataset in the TSV format and configuring the model (e.g., using a linear layer for pose data and mT5 as the backbone), the original results were reproduced without additional code. The framework then allowed easy experimentation with different input modalities, such as pre-computed I3D features from RGB frames and end-to-end video-to-text translation, demonstrating performance improvements.
- Machine Translation From Pixels: For languages with complex scripts, like Hebrew, standard tokenization can be inefficient. MultimodalHugs enabled a novel approach where each ‘word’ was encoded as an image. The framework treated these image sequences like video, embedding each ‘frame’ using CLIP. This setup allowed fine-tuning an M2M-100 model for Hebrew-to-English translation without modifying the model code. The pixel-based model performed on par with the original, but required 45% fewer tokens, which is beneficial for memory-constrained or large-context applications. Manual evaluation also indicated better translation of named entities and misspellings.
Also Read:
- SignClip: A New Framework for Accurate Sign Language Translation Using Gestures and Mouthing
- Decoupling Sensor Knowledge: A New Approach for Efficient AI Perception
Looking Ahead
MultimodalHugs represents a significant step forward for Sign Language Processing and broader multimodal research. By providing general-purpose features like modality-specific processors, a standardized dataset interface, and a flexible training pipeline, it lowers entry barriers and promotes reproducible research in underrepresented domains. The framework’s open-source nature (https://github.com/GerrySant/multimodalhugs) encourages community contributions to further advance SLP and multimodal research.
Future work plans to support mixed-modality input sequences, allowing users to freely interleave textual and non-textual signals within a single input, further enhancing the framework’s flexibility and alignment with modern multimodal language models.
