TLDR: A study on zero-shot classification of human postures (sitting, standing, walking/running) using Vision-Language Models (VLMs) found that for high-performing models like MetaCLIP 2 and OpenCLIP, the simplest, most basic text prompts yielded the best results, with added detail degrading performance (“prompt overfitting”). Conversely, a lower-performing model (SigLip) benefited from more descriptive, body-cue-based prompts for ambiguous classes. The research suggests preferring label-style prompts as a default and using geometric descriptions selectively for ambiguous categories in data-scarce scenarios.
In the rapidly evolving field of artificial intelligence, particularly in computer vision, a significant challenge remains: recognizing human actions and postures from images when there’s very little labeled data available. Traditional methods often require vast datasets for training, which can be expensive and time-consuming to acquire. This is where Vision-Language Models (VLMs) offer a promising solution, enabling what’s known as “zero-shot classification” – identifying objects or actions without explicit prior training on those specific categories.
A recent research paper, titled “Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcity,” explores how the way we phrase text prompts influences the ability of these advanced VLMs to classify common human postures like sitting, standing, and walking/running. The study, conducted by MingZe Tang and Jubal Chandy Jacob from the University of Aberdeen, delves into a counter-intuitive finding that could reshape how we interact with these powerful models.
The Challenge of Data Scarcity and Zero-Shot Learning
Human action recognition from still images is crucial for many applications, but obtaining balanced and annotated datasets is a major hurdle. VLMs, like OpenCLIP, MetaCLIP 2, and SigLip, address this by aligning images and text in a shared understanding space. This allows text descriptions to act as labels during inference, even for categories the model hasn’t seen during training. The core question the researchers tackled was whether carefully chosen wording for these text labels could improve classification accuracy, especially when data is scarce.
Investigating Prompt Specificity
The study used a small dataset of 285 images derived from COCO, focusing on three everyday postures: sitting, standing, and walking/running. The researchers evaluated a suite of modern VLMs, alongside unimodal vision models (DINOv3, Vision Transformer) and a pose-centric structural model (YOLOv11x-pose). The key experimental factor was prompt specificity, which was systematically varied across three tiers:
- Tier 1 (Minimal Label): Simple prompts like “a photo of a person [class]” (e.g., “a photo of a person sitting”).
- Tier 2 (Action Cue): Added a brief action description, such as “a person seated on a chair” or “a person standing still and upright.”
- Tier 3 (Anatomical/Pose Constraints): Incorporated compact pose geometry, for instance, “hips and knees bent at right angles” for sitting or “legs straight and torso vertical” for standing.
Crucially, prompts excluded scene, identity, and clothing terms to ensure that any observed differences in performance were solely due to the pose description.
Surprising Findings: Simplicity Often Wins
The results revealed a compelling and often counter-intuitive trend. For the highest-performing VLMs, MetaCLIP 2 and OpenCLIP, the simplest, most basic Tier 1 prompts consistently achieved the best classification results. Adding more descriptive detail in Tier 2 or anatomical cues in Tier 3 significantly degraded their performance. For example, MetaCLIP 2’s multi-class accuracy dropped from 68.8% with a Tier 1 prompt to 55.1% with a Tier 2 prompt. The researchers termed this phenomenon “prompt overfitting,” suggesting that excessive detail can unduly constrain these powerful models and hinder their ability to generalize.
Conversely, the lower-performing SigLip model showed a different response. While its overall accuracy remained lower, it demonstrated improved classification for ambiguous classes, particularly “walking/running,” when given more descriptive, body-cue-based Tier 3 prompts. This highlights that the optimal prompt strategy can be highly model-dependent.
The study also compared these VLMs to other models. Models with semantic (VLMs) or structural (YOLOv11x-pose) understanding demonstrated a clear performance advantage over unimodal models that learn from pixels alone, such as the standard Vision Transformer (ViT) and DINOv3.
Why Prompt Wording Matters
The researchers discuss that minimal, noun-centric prompts (Tier 1) align well with how VLMs are pre-trained on vast image-text pairs, where concept names are frequent and broadly understood. Adding action cues (Tier 2) can introduce a linguistic-visual mismatch for still images, as verbs like “walking” imply dynamics or intent rather than a static appearance. This can reduce the model’s confidence and increase confusion between similar classes.
However, geometric phrasing (Tier 3) can be beneficial for visually ambiguous categories. These prompts specify local, view-stable relations (e.g., angles at joints) that are directly verifiable in a single image. Visualizations showed that geometric wording encouraged models to focus attention more on relevant body regions and less on background elements.
Also Read:
- Unpacking CLIP’s Edge: Language Supervision vs. Data Scale in Vision Encoders
- ToMCLIP: Enhancing Multilingual Vision-Language Models Through Topological Alignment
Practical Takeaways for Low-Resource Settings
Based on their findings, the authors propose a simple policy for deploying VLMs in situations with limited data:
- As a default, prefer label-style prompts (Tier 1) for each class.
- If confusions persist for visually similar categories, selectively introduce compact geometric descriptions for those specific classes.
- Be cautious with action verbs for single images, as they may not be consistently grounded.
- If language supervision is unavailable, a pose estimation pipeline (like YOLOv11-Pose with geometric rules) can be a viable alternative, provided keypoint detection is reliable.
This research provides valuable insights into optimizing zero-shot classification for human postures under data scarcity, emphasizing that sometimes, less is indeed more when it comes to prompt engineering for high-performing Vision-Language Models. You can read the full paper here.
