TLDR: Researchers developed an automatic sign language recognition system using a hybrid CNN-LSTM deep learning architecture and Mediapipe for gesture keypoint extraction. The Python-based system provides real-time translation with 92% average accuracy, showing strong performance for distinct gestures but facing challenges with visually similar ones. It holds significant potential for applications in healthcare, education, and public services to enhance accessibility for deaf communities.
Sign languages are fundamental for communication within deaf communities worldwide, yet they often face marginalization, which can severely limit access to crucial services like healthcare and education. Addressing this challenge, a new study introduces an innovative automatic sign language recognition system designed to bridge these communication gaps.
The research, led by Takouchouang Fraisse Sacré from the Department of Computer Science at the National University of Vietnam, proposes a system built upon a hybrid architecture combining Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks. A key component of this system is the use of Mediapipe, a Google library, for the precise extraction of keypoints from gestures.
The primary goal of this project is to develop a system capable of automatically translating sign gestures into text or voice in real-time. This is achieved by leveraging recent advancements in computer vision and deep learning.
How the System Works
At its core, the system utilizes Mediapipe to capture detailed information about a person’s movements. For every image captured by a camera, Mediapipe identifies:
- 21 keypoints for each hand, mapping articulations and fingertips.
- 468 keypoints for the face, defining contours and expressions.
- 33 keypoints for the body, indicating the position of major joints.
These spatial coordinates (x, y, z) provide a rich representation of gestures, encompassing not only hand configurations but also crucial facial expressions and body movements integral to sign languages.
The extracted keypoint data then feeds into a hybrid CNN-LSTM model. The CNN component is responsible for extracting spatial features from the captured images, using multiple convolutional layers to identify patterns at different scales. Following this, the LSTM component models the temporal dependencies of gestures over time. LSTMs are particularly effective for sequence recognition, capable of remembering information over long periods and adapting to variations in gesture execution speed. This powerful combination allows the system to analyze both the spatial appearance and the temporal flow of gestures for highly accurate recognition.
Development and Performance
The system was developed using Python, with TensorFlow for building and training the deep learning model, Mediapipe for keypoint detection, Streamlit for an interactive, real-time user interface, and OpenCV for image processing and camera access.
During testing, the model achieved an impressive average accuracy of 92%, with a recall of 89% and an F1-score of 90.5%. It demonstrated very strong performance for distinct gestures, such as “Hello” and “Thank you.” However, the study also highlighted that some confusions still occur for visually similar gestures, like “Call” and “Yes.” Despite these challenges, the model shows a good balance between performance and computational efficiency, offering competitive accuracy with lower inference times compared to some other state-of-the-art approaches.
Also Read:
- Vision Language Models Advance Human Activity Recognition in Healthcare
- Turning Tables into Touch Interfaces: Energy-Efficient AI for Gesture Recognition
Future Directions and Applications
While the results are promising, the researchers acknowledge limitations, including difficulties with similar gestures, dependence on lighting conditions, and variations in gesture execution among individuals. Future improvements will focus on expanding data augmentation, exploring more advanced neural network architectures like Transformers, refining data preprocessing techniques, and integrating contextual and linguistic information to recognize complete sentences.
The potential applications of this system are vast and impactful:
- Healthcare: Facilitating communication between deaf patients and medical staff, thereby improving access to care and consultation quality.
- Education: Serving as a pedagogical tool for learning sign languages for both deaf and hearing individuals, and aiding the integration of deaf students in mainstream education.
- Public Services: Enhancing accessibility in public administrations, transport, and other services, contributing to a more inclusive society.
The complete implementation of the system, including preprocessing scripts, training code, and the Streamlit user interface, is openly available on GitHub, allowing for reproduction and further development. You can find more details about this research paper here: Automatic Sign Language Recognition: A Hybrid CNN-LSTM Approach Based on Mediapipe.
This project marks a significant step towards making communication more accessible for deaf communities, with ongoing work aimed at enhancing the model’s robustness and expanding its recognized vocabulary for diverse real-world applications.
