TLDR: AulSign is a novel method that leverages Large Language Models (LLMs) to achieve accurate sign language translation, particularly in low-resource environments. It addresses the challenge of data scarcity by using dynamic prompting, in-context learning with sample selection, and associating signs with natural language descriptions. Comprising a Retriever, an LLM, and a Sign Mapper, AulSign has demonstrated superior performance over state-of-the-art models in both spoken-to-sign and sign-to-spoken tasks for American Sign Language (ASL) and Italian Sign Language (LIS), significantly enhancing accessibility and inclusivity for the Deaf community.
Translating natural languages into sign languages is a complex and often overlooked challenge. Despite growing interest in making communication more accessible, developing effective translation systems has been difficult due to a scarcity of data that aligns natural language with sign language. Current methods often struggle in these data-poor environments because the few available datasets are usually very specific, lack consistent standards, or don’t fully capture the rich linguistic details of sign languages.
To tackle this significant limitation, researchers Luana Bulla, Gabriele Tuccio, Misael Mongiovì, and Aldo Gangemi have introduced a novel method called Advanced Use of LLMs for Sign Language Translation, or AulSign. This innovative approach leverages the power of Large Language Models (LLMs) through dynamic prompting and in-context learning, combined with careful sample selection and sign association. While LLMs are incredibly skilled at processing text, they don’t inherently understand sign languages. AulSign overcomes this by associating signs with concise natural language descriptions and instructing the LLM to use these descriptions for translation.
The AulSign method is built around three main components: a Retriever, an LLM, and a Sign Mapper. The Retriever module is responsible for identifying and retrieving relevant examples from a training set. These examples are then used to guide the LLM in converting an input sentence into a ‘pseudo-language’ – a sequence of clear, unambiguous descriptions of signs. This set of examples, along with grammatical rules, is integrated into the prompt given to the LLM, providing it with a comprehensive linguistic and structural context.
For spoken-to-sign translation, the LLM generates this pseudo-language sequence, which the Sign Mapper then converts into the target sign language by matching each part of the sequence to an entry in a predefined lexicon. The reverse process is used for sign-to-spoken translation. AulSign can handle languages not well-represented in typical LLM training data by incorporating external vocabularies and structured linguistic representations.
The researchers evaluated AulSign using both English and Italian languages, employing recognized benchmarks like SignBank+ for American Sign Language (ASL) and the Italian LaCAM CNR-ISTC dataset for Italian Sign Language (LIS). The results demonstrated AulSign’s superior performance compared to state-of-the-art models, particularly in low-data scenarios. For instance, in low-resource ASL spoken-to-sign translation, AulSign showed a significant improvement in F1 score over the baseline model. Similarly, for Italian LIS, AulSign achieved substantial gains across all evaluation metrics, including F1-score, BLEU, ChrF2, and even positional accuracy (MAE).
A key aspect of AulSign’s success is its use of Formal SignWriting (FSW) as an intermediate representation. FSW provides a standardized, linearized format for encoding sign sequences, capturing essential features like handshapes, orientations, movements, and body locations. This notation is not only compact and linguistically grounded but also implicitly explainable, making it valuable for computational processing and widely adopted within the Deaf community.
The modular architecture of AulSign also ensures transparency throughout the translation pipeline. This means that users and researchers can understand how translations are generated, identify potential errors, and trace the steps of inference, which is often not possible with opaque end-to-end models. This explainability helps build trust and allows for targeted improvements.
While AulSign shows promising results, especially in low-resource settings, the authors acknowledge some limitations. The model’s reliance on a specialized vocabulary of canonical descriptions introduces a degree of rigidity, and its translation quality is still dependent on the availability and consistency of training data. However, the approach is designed to be extensible to other notational systems like HamNoSys or SMPL-X, offering avenues for future adaptability.
Also Read:
- Unlocking LLM Potential: A Seed-Free Approach to Instruction Tuning
- Unlocking Speed in Video LLMs: Verifier-Guided Token Pruning for Faster Decoding
In conclusion, AulSign represents a significant step forward in sign language translation, leveraging retrieval-augmented generative models to enable translation for previously unseen languages. Its ability to excel in low-resource scenarios for both ASL and LIS, coupled with its explainable architecture, holds immense potential to enhance accessibility and inclusivity in communication technologies for underrepresented linguistic communities. You can read the full research paper here.
