TLDR: Pose-Star is a novel framework for fashion image editing that addresses limitations of existing methods by dynamically generating anatomy-aware masks. It uses pose-guided calibration of diffusion attention for precise localization of body structures, a multi-phase filtering process to aggregate noise-free regions, and an edge optimization module for sharp boundaries. This enables flexible, pose-robust, and generalizable editing, compatible with various existing image editors for applications like virtual try-on and style customization.
Fashion image editing is a rapidly evolving field with significant applications in virtual try-on and clothing design. However, existing methods often struggle with real-world adaptability, facing challenges such as limited user-defined region flexibility, poor robustness to complex poses, and difficulties in generalizing to diverse, cluttered scenes.
Current fashion-specific editing pipelines typically involve two stages: generating a mask for the editing region, followed by diffusion-based image manipulation. A major drawback of these approaches is their over-reliance on generator optimization while neglecting the crucial aspect of mask controllability. This leads to two critical limitations: first, poor user flexibility, where coarse masks restrict edits to broad areas like the upper torso, and fine-grained masks, while preserving poses, prevent style or length customization. Second, weak pose robustness, as mask generators often fail with articulated poses and miss rare body regions like the waist, while human parsers are limited by predefined categories.
Also Read:
- Unlocking Domain-Generalizable Portrait Style Transfer
- CorrDetail: Enhancing Deepfake Detection with Self-Correction and Visual Detail
Introducing Pose-Star: A New Paradigm for Fashion Image Editing
To address these significant gaps, researchers have introduced Pose-Star, a groundbreaking framework designed to dynamically recompose body structures into anatomy-aware masks for highly flexible, user-defined edits. Pose-Star is a plug-and-play, training-free solution that combines the style adaptability of coarse human masks with the precision of fine-grained clothing masks. It achieves this by dynamically generating detailed human masks guided by user instructions, such as “belly-length.”
The core innovation of Pose-Star lies in its ability to decompose anatomical prompts into local body structures (e.g., “belly-length” translates to neck, chest, waist, etc.) and then perform precise localization, aggregation, and optimization to create the target human mask. The framework is built upon three key modules:
- Localization: Pose-Guided Keypoint-Star
This module enhances the precision of locating rare anatomical regions, even in complex poses. It calibrates diffusion-derived attention tokens (called Star tokens) using skeletal keypoints from OpenPose. This ensures robust localization by aligning token maps with biomechanically plausible positions, overcoming the limitations of traditional open-world detectors that struggle with fine-grained or articulated body parts.
- Aggregation: Multi-Phase Token Filtering
To synthesize user-specified anatomical regions while suppressing noise, Pose-Star analyzes the dynamic evolution of diffusion attention maps across different steps. It identifies three phases: Convergence (attention focuses on target regions), Stabilization (attention is steadily focused), and Divergence (attention becomes unstable). By applying threshold masking and sliding-window fusion, the system preserves stable regions from the stabilization phase and eliminates noise from the other phases, ensuring robust anatomical localization.
- Optimization: Edge-Aware Selection
This module refines the mask boundaries to enhance sharpness and preserve anatomical topology. It merges cross-attention maps (which localize fine-grained anatomy but can be blurry) with self-attention maps (which capture sharp edges but lack anatomical specificity). This synergistic approach, combined with Canny edge alignment, ensures that edited regions respect both anatomical structure and pose geometry, maintaining boundary-pose consistency.
Pose-Star is highly compatible with existing mask-guided editors, including virtual try-on models like IDM-VTON and inpainting models like PowerPaint, without requiring fine-tuning. This seamless integration significantly enhances their capabilities, allowing for unprecedented functionality such as customizable clothes length adjustments, precise edits on highly articulated poses, and robust adaptation to wild scenarios with cluttered backgrounds. It even extends to video editing applications, seamlessly modifying dance sequences with complex poses and dynamic camera angles.
Quantitative evaluations demonstrate that Pose-Star significantly improves user-defined region flexibility, pose robustness, and in-the-wild generalizability compared to state-of-the-art fashion-specific and general-purpose image editors. While diffusion inversion incurs some computational overhead, subsequent anatomical edits are remarkably fast due to optimized token aggregation.
Despite its advancements, Pose-Star has some limitations, such as struggling with transparent occlusions where body parts are obscured by glass or wire mesh, and issues with preserving original facial identity during cross-garment category edits when the face is initially occluded. These areas are identified for future research to further enhance the framework’s capabilities.
For more technical details, you can refer to the full research paper: Pose-Star: Anatomy-Aware Editing for Open-World Fashion Images.
