TLDR: HMVLM is a new AI framework that integrates 3D human motion, vision, and language understanding. It uses a Mixture of Expert Low-Rank Adaption (MoE LoRA) strategy with a “zero expert” to prevent catastrophic forgetting of linguistic knowledge during multimodal training. Additionally, it employs body-part-specific tokenization for more detailed and accurate human pose representation. The model demonstrates strong performance across text-to-motion generation, pose estimation, and motion video understanding, while effectively preserving its foundational language capabilities.
Researchers have introduced a new framework called HMVLM, which stands for Human Motion-Vision-Language Model. This innovative system aims to bridge the gap between understanding human motion, visual information, and language, addressing some significant challenges in the field of artificial intelligence.
One of the primary hurdles in combining human motion data with powerful language models is what scientists call “catastrophic forgetting.” This happens when a model, after learning new information (like human motion), tends to forget its previously acquired knowledge, such as general language understanding. HMVLM tackles this by using a clever strategy known as Mixture of Expert Low-Rank Adaption (MoE LoRA).
The MoE LoRA approach allows the model to dynamically assign different “experts” or specialized components based on the input it receives. For instance, if the input is a general language query, the system can route it to a “zero expert.” This zero expert is designed to preserve the model’s original language capabilities, effectively preventing it from forgetting what it already knows about language. When the task involves human motion, other specialized experts are engaged, ensuring that the model can adapt to new tasks without compromising its foundational knowledge.
Another key innovation in HMVLM is its approach to representing human poses. Traditional methods often struggle to capture the detailed spatial information of a pose. HMVLM overcomes this by breaking down the human body into different parts, like limbs and torso. Each body part is then tokenized separately, similar to how words are broken into tokens in language processing. This “body-part-specific tokenization” significantly enhances the resolution and expressiveness of how the model understands and represents human motion, making it more accurate for tasks like pose estimation.
The HMVLM framework is designed to be versatile, supporting a wide array of human-centric tasks. These include generating human motion from text descriptions (text-to-motion synthesis), estimating human poses from images, and understanding human actions in videos. Experiments have shown that HMVLM not only effectively reduces knowledge forgetting during its training but also achieves impressive performance across these diverse tasks.
For example, in evaluations of dialogue abilities, HMVLM demonstrated only a minimal degradation in performance after being fine-tuned for motion tasks, unlike other methods that suffered severe drops. In text-to-motion generation, it showed strong results compared to existing state-of-the-art models. Its body-part-based tokenization also proved superior in capturing limb details for pose estimation, leading to more accurate predictions.
Also Read:
- Preventing AI Forgetting: A New Routing Approach for Vision-Language Models
- ToolScope: Enhancing Multimodal AI Agents with Strategic Planning and Dynamic Visual Perception
This research marks a significant step towards creating more robust and adaptable multimodal AI systems that can seamlessly integrate complex human motion data with language and vision. While the model shows great promise, the researchers acknowledge that future work will focus on achieving more holistic integration between modalities and improving its ability to generate any-to-any motion across different datasets. You can read the full research paper for more technical details and results here.
