TLDR: This research paper compares encoder-based and decoder-based video foundation models for classifying breakdance moves (powermoves, footwork, toprock) using the BRACE dataset. It concludes that encoder models, particularly ImageBind, generally outperform decoder-based Video Language Models like Qwen2.5-VL for predictive tasks. The study also provides insights into decoder finetuning, suggesting that non-greedy decoding and additional label descriptions can improve performance for generative models when applied to prediction.
Artificial intelligence has made significant strides in various domains, from social media to healthcare. However, its application in sports, particularly for accurate identification and natural language description of actions, has largely focused on popular sports like soccer or basketball. A recent research paper delves into a niche yet globally popular dance sport: breakdance, analyzing how modern video foundation models can classify its intricate moves.
The paper, titled “Breakdance Video classification in the age of Generative AI,” explores the effectiveness of both encoder and decoder-based video models for breakdance move classification. This is particularly relevant as breakdancing made its debut as an Olympic sport in Paris 2024, sharing spatio-temporal characteristics with other high-speed, intermittent sports.
The BRACE Dataset: A Foundation for Analysis
At the heart of this research is the BRACE dataset, a collection of videos from the Red Bull BC One breakdancing competition. This dataset features top dancers in a 1-vs-1 battle format, providing a rich source of diverse breakdance sequences. For this study, the task was to recognize specific dance moves from video streams. The three primary moves labeled are:
- Powermoves: Dynamic, acrobatic movements involving continuous rotation across body parts, often perceived as the most impressive.
- Footwork: Fundamental ground-based movements where dancers use their legs in intricate patterns, supported by their hands. Also known as “downrock” or “floorwork.”
- Toprock: The upright, standing dance style performed in breaking, serving as an entry to floor moves, showcasing rhythm and coordination.
Additionally, segments with no specific moves are labeled as ‘None’. The models predict these moves within 10-second video segments, with an overlapping stride of 5 seconds, a window size chosen to balance context and avoid excessive overlap.
Encoder vs. Decoder Models: A Comparative Study
The researchers compared two main categories of video foundation models:
- Encoder-based Models: These models process video by tokenizing spatial and temporal patches and passing them through a transformer architecture. They are traditionally favored for predictive tasks like video classification. The study utilized state-of-the-art encoders such as ViViT (Video Vision Transformer), Video MAE (Video Masked Autoencoder), and ImageBind.
- Decoder-based Models: These are modern Video Large Language Models (VLMs) that combine a video processing component with a language model to interpret visual information and respond in a human-like way. For this research, Qwen2.5-VL was used as a representative decoder model.
The findings revealed a consistent trend: encoder-based models generally delivered superior performance for breakdance move prediction compared to their decoder-based counterparts. While large decoder models often benefit from extensive pretraining, their fine-tuning is typically geared towards token generation, which may not translate as effectively to direct prediction tasks.
Insights from Encoder Embeddings and Decoder Finetuning
Further analysis into the encoder models showed that ImageBind embeddings provided better generalization and separability between different dance moves. This is likely because ImageBind is designed to capture generic semantic relationships across various modalities (images, text, videos), making it more adaptable to new tasks compared to ViViT or Video MAE, which are often pretrained for specific tasks like classifying Kinetics-400 videos.
The study also conducted a detailed ablation study on decoder fine-tuning, exploring several design parameters for Qwen2.5-VL:
- LoRA Rank: The complexity of the model, influenced by the LoRA rank, significantly impacted generalization. Lower ranks struggled with more complex descriptions, while higher ranks stabilized performance.
- Greedy vs. Non-Greedy Decoding: Surprisingly, for predictive tasks, non-greedy decoding (with a higher temperature parameter) combined with sufficient model complexity (higher LoRA rank) was preferred over greedy decoding for better factual accuracy. Greedy decoding, while deterministic, didn’t necessarily lead to more accurate predictions in this context.
- Additional Label Descriptions: Including detailed descriptions of the dance moves in the labels during fine-tuning improved generalization for sufficiently complex models. This acts as a form of regularization, adding ‘noise’ to the decision probability space and contributing to better predictive performance.
Also Read:
- Pinpointing Evidence: A New Approach to Video Understanding
- Evaluating Neural Embeddings for Earth Observation: The NeuCo-Bench Framework
Conclusion: A Path Forward for Niche Sports AI
This research provides valuable insights into applying advanced AI video models to niche sports like breakdancing. It confirms that for predictive tasks, encoder models still hold an advantage over decoder-based VLMs. Furthermore, it highlights the importance of choosing models with general semantic understanding capabilities, like ImageBind, for better task transfer. The detailed ablation study on decoder fine-tuning offers practical guidance for improving the predictive performance of generative models, suggesting that techniques like non-greedy decoding and enriched label descriptions can act as effective regularization. This work paves the way for more accurate and nuanced AI analysis in a broader range of sports. You can read the full research paper here.
