TLDR: H2OT (Hierarchical Hourglass Tokenizer) is a new framework designed to make 3D human pose estimation from videos more efficient. It tackles the high computational costs of Video Pose Transformers (VPTs) by intelligently pruning redundant pose tokens in a hierarchical manner during intermediate processing stages and then recovering the full-length sequence for output. This ‘plug-and-play’ method significantly reduces computational resources (FLOPs, GPU memory, training time) and boosts inference speed (FPS) while maintaining or improving accuracy, making VPTs more practical for resource-constrained environments.
Estimating 3D human poses from videos is a crucial task with applications ranging from action recognition to human-robot interaction. However, a major challenge in this field, especially for advanced transformer-based models known as Video Pose Transformers (VPTs), is their high computational cost. These models often process very long video sequences, leading to significant resource demands that make them impractical for devices with limited processing power.
Addressing this challenge, researchers Wenhao Li, Mengyuan Liu, Hong Liu, Pichao Wang, Shijian Lu, and Nicu Sebe have introduced a novel framework called H2OT: Hierarchical Hourglass Tokenizer. This innovative approach aims to make 3D human pose estimation from videos much more efficient without compromising accuracy. The core idea behind H2OT is to intelligently prune redundant information within video frames and then reconstruct the full sequence when needed.
Understanding H2OT’s Approach
Traditional VPTs typically maintain the full-length sequence of pose tokens (each representing a video frame) throughout all their processing layers. This “rectangle” paradigm, while effective, leads to expensive and often redundant calculations. H2OT, on the other hand, adopts a “trophy-shaped” or “pyramidal” paradigm. It starts by processing the full sequence, then progressively prunes (removes) pose tokens from redundant frames in the intermediate transformer blocks, and finally recovers the full-length sequence at the end. This means that the computationally intensive intermediate layers work with significantly fewer tokens, drastically improving efficiency.
The framework consists of two key modules:
- Token Pruning Module (TPM): This module dynamically selects a few representative tokens, effectively eliminating the redundancy present in video frames. Unlike previous methods that might prune tokens in one large chunk, H2OT introduces a hierarchical pruning strategy. This means the number of tokens is gradually reduced across network layers, creating a pyramidal feature hierarchy that helps preserve more useful information while reducing computational load. For efficient and parameter-free pruning, H2OT primarily uses a Token Pruning Sampler (TPS), which employs a linear sampling strategy to select tokens.
- Token Recovering Module (TRM): After the pruning, the network operates with a reduced number of tokens. The TRM is responsible for restoring the detailed spatio-temporal information based on these selected tokens, expanding the network’s output back to the original full-length temporal resolution. This is crucial for real-world 3D human pose estimation systems that need to predict poses for all frames. H2OT utilizes a Token Recovering Interpolation (TRI) module, a simple and efficient interpolation operation, to achieve this.
Also Read:
- EHWGesture: A Comprehensive Resource for Understanding Clinical Hand Movements
- Unlocking YOLOv10s Speed on Standard Hardware
Key Innovations and Benefits
H2OT is designed as a “plug-and-play” framework, meaning it can be easily integrated into existing VPT models with minimal modifications. It supports both common inference pipelines: “seq2seq” (outputting 3D poses for all frames) and “seq2frame” (outputting the 3D pose of a center frame). The research demonstrates that maintaining the full pose sequence throughout the entire process is often unnecessary; a few representative pose tokens can achieve both high efficiency and accurate estimation.
Compared to previous work, including the authors’ own conference paper (HoT), H2OT introduces several significant improvements:
- A novel hierarchical pruning strategy that gradually reduces tokens, leading to a more effective reduction of video redundancy.
- The adoption of parameter-free and fast sampling pruning (TPS) and interpolation recovering (TRI) strategies, which address the additional parameter and inference time burdens of earlier cluster-based or attention-based methods.
Extensive experiments on benchmark datasets like Human3.6M and MPI-INF-3DHP show that H2OT significantly reduces computational costs (FLOPs, GPU memory, training time) and increases inference speed (FPS) while maintaining or even improving pose estimation accuracy. For instance, when applied to MixSTE, H2OT reduced FLOPs by 57.4% and improved FPS by 87.8%, with a 0.5mm improvement in MPJPE.
This framework represents a significant step towards making advanced 3D human pose estimation models more practical and deployable on resource-constrained devices, paving the way for stronger and faster Video Pose Transformers. You can read the full research paper here: H2OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers.
