TLDR: FAME is a new training-free video editing framework that tackles gender stereotypes in profession-related prompts. It achieves this by softly injecting debiasing tokens into the text encoder and integrating fairness modulation into both temporal self-attention and prompt-to-region cross-attention. This approach mitigates bias, preserves prompt alignment, and maintains temporal consistency, outperforming existing methods on a new fairness-oriented video editing benchmark called FairVE.
In the rapidly evolving world of AI-powered video editing, a significant challenge has emerged: the tendency of current models to perpetuate gender stereotypes, especially when generating content related to professions. For instance, a prompt like “a teacher is playing tennis” might default to a male figure, complete with stereotypical attire, due to biases embedded in the training data of these models. This issue not only reflects societal biases but also leads to visual inconsistencies and motion corruption when attempts are made to debias the content directly.
Addressing this critical gap, researchers have introduced FAME: Fairness-aware Attention-modulated Video Editing. This innovative, training-free framework is designed to mitigate profession-related gender biases while ensuring that the edited videos remain aligned with the original prompt and maintain temporal consistency, resulting in coherent and high-quality video edits.
The Problem with Existing Video Editing Models
Current training-free video editing (VE) models, often built upon text-to-image (T2I) Diffusion Models like Stable Diffusion, have made great strides in controlling object appearance, motion, and semantics. However, they frequently reproduce gender and racial stereotypes. This bias largely stems from imbalanced image-text datasets and inherent biases in large-scale language models used for text encoding. While fairness-aware T2I methods exist for static images, they often fail when applied to videos, leading to issues like semantic mismatch, visual inconsistency, and degraded controllability across frames.
How FAME Tackles Bias and Motion Corruption
FAME introduces a novel approach that integrates fairness control without compromising motion quality. It operates at inference time, meaning no expensive retraining of the entire model is required. The framework consists of three key components:
-
Soft Debiasing Prompt Encoding: Instead of simply adding debiasing terms, FAME gently adjusts the prompt embeddings. It identifies fairness-sensitive tokens within a target prompt (e.g., “teacher”) and softly fuses their embeddings with corresponding fairness attributes (e.g., “female”). This process also refines the End of Sequence (EOS) token embedding, which aggregates global prompt semantics, to inject fairness information without disrupting local word-level alignments. This ensures fairness cues are subtly woven into the prompt’s meaning.
-
Temporal Self-Attention Modulation: To maintain fairness consistency across video frames and prevent fairness cues from fading over time, FAME enhances the temporal self-attention mechanism. It uses a region-constrained attention mask combined with time decay weighting. This mechanism amplifies attention within fairness-sensitive regions (e.g., a person’s outfit) while suppressing irrelevant interactions between different regions, thereby preserving intra-region coherence across frames.
-
Cross-Attention Reweighting: Fairness tokens can sometimes drift from their correct visual regions during the denoising process. FAME addresses this by reweighting token-to-region matching scores in the cross-attention mechanism. It incorporates fairness-sensitive similarity masks derived from debiasing prompt embeddings. This ensures that fairness-sensitive semantics remain tied to the right visual regions and prevents temporal drift, leading to better alignment between textual concepts and visual elements across frames.
Introducing FairVE: A New Benchmark for Fairness in Video Editing
To systematically evaluate fairness in video editing, the researchers also developed FairVE, the first benchmark dataset tailored for this purpose. FairVE comprises video-text pairs sourced from existing datasets and user-generated content, focusing on scenes with a single human subject to capture naturally occurring social biases. It specifically evaluates gender-profession stereotypes across eight representative professions (e.g., CEO, doctor, nurse) by combining them with gender-specific terms to create a comprehensive set of prompts.
Also Read:
- Video-As-Prompt: A Unified Framework for Semantic Video Generation
- VESSA: Adapting Vision Models with Object-Centric Videos
Experimental Success and Future Directions
Extensive experiments on the FairVE benchmark demonstrate that FAME consistently outperforms state-of-the-art training-free video editing models in fairness preservation, semantic accuracy, and visual consistency. Automatic metrics, such as Correction Count and Correction Ratio, show clear improvements in reducing gender bias, especially for strongly stereotyped professions. Human evaluations further confirm FAME’s effectiveness in perceived bias correction, semantic relevance, and visual stability.
FAME also achieves superior performance across key CLIP-based metrics (CLIP-F, CLIP-T, TIFA) and exhibits the lowest warp error, indicating better temporal consistency and spatial coherence in edited sequences. The ablation studies confirm the crucial role of each module, showing that combining soft prompt encoding, self-attention modulation, and cross-attention reweighting progressively enhances both video editing quality and fairness.
By working entirely at inference time, FAME offers an efficient and effective solution to a pressing problem in AI ethics, paving the way for more responsible and unbiased video generation. For more details, you can read the full research paper here.
