TLDR: AEGIS is a new large-scale benchmark dataset of over 10,000 real and AI-generated videos, including highly realistic ones created with advanced models and GPT-4o refined prompts. It aims to improve AI-generated video detection by providing diverse, complex, and challenging content with rich multimodal annotations. Initial tests show current state-of-the-art models struggle significantly with AEGIS’s most difficult synthetic videos, highlighting the need for more robust detection methods.
The rapid advancement of AI in generating highly realistic video content has brought about exciting new possibilities, but also significant challenges, particularly concerning the authenticity of digital media. As AI-generated videos become increasingly sophisticated, distinguishing them from real footage becomes a critical task for maintaining societal trust and digital integrity. A new research paper introduces AEGIS, a groundbreaking benchmark designed to tackle this very issue.
AEGIS, which stands for Authenticity Evaluation Benchmark for AI-Generated Video Sequences, is a novel, large-scale dataset specifically created to evaluate how well current AI models can detect hyper-realistic and semantically complex AI-generated videos. Traditional benchmarks often fall short due to limited realism, insufficient scale, or a lack of complexity, making them inadequate for assessing modern vision-language models against advanced forgeries.
Developed by Jieyu Li, Xin Zhang, and Joey Tianyi Zhou, AEGIS comprises over 10,000 meticulously curated real and synthetic videos. These synthetic videos are generated using a diverse range of state-of-the-art generative models, including well-known open-source tools like Stable Video Diffusion, CogVideoX-5B, and I2VGen-XL, as well as proprietary systems such as KLing and Sora. This broad spectrum of generative sources ensures that AEGIS represents the cutting edge of AI video creation, making it a truly challenging benchmark.
One of the key innovations of AEGIS is its inclusion of specially constructed “challenging subsets.” These subsets feature videos created with prompts refined by advanced AI models like GPT-4o, resulting in scenarios that are unprecedentedly realistic and difficult to distinguish from genuine footage. This rigorous approach is crucial for evaluating the robustness of detection models.
Beyond just video content, AEGIS also provides rich multimodal annotations. These include Semantic-Authenticity Descriptions (semantic summaries, generation metadata, and reasoning explanations), Motion Features (like optical flow), and Low-level Visual Features (such as frequency-domain analysis). These annotations are invaluable for not only detecting authenticity but also for supporting deeper analysis, such as understanding why a video might be flagged as fake or for localizing forged elements within a video.
The construction of the AEGIS dataset involved a systematic pipeline. Real videos were collected from diverse sources like the Vript Dataset, DVF Dataset, and supplemental YouTube collections, ensuring high realism and visual complexity. Synthetic videos were sourced from the TIP-I2V dataset and independently generated using proprietary models like KLing and Sora, with prompts carefully refined by GPT-4o to enhance detail and realism.
A crucial step in creating AEGIS was its rigorous filtering process, guided by three principles: Reality, Difficulty, and Diversity. The “Reality” principle ensured that all videos were photorealistic, removing any art-style or low-quality content. The “Difficulty” principle involved screening out synthetic videos that were too easy to detect, ensuring that the remaining samples posed a genuine challenge. Finally, “Diversity” ensured a wide range of scene types, subjects, visual conditions, durations, and resolutions in both real and synthetic videos.
The dataset is strategically split into Training, Validation, and a particularly challenging Hard Test Set. The Hard Test Set is designed to assess model robustness and generalization against sophisticated forgeries, featuring complex authentic videos and advanced synthetic content from models like KLing and Sora.
Extensive experiments were conducted using advanced vision-language models like Qwen2.5-VL and Video-LLaVA. The results highlighted a significant finding: these state-of-the-art models demonstrated limited detection capabilities, especially on the most challenging subsets of AEGIS. For instance, under zero-shot settings, models showed very low accuracy in detecting synthetic videos on the Hard Test Set. Even with structured reasoning prompts or fine-tuning, while some improvements were noted on in-domain data, generalization to the Hard Test Set remained a persistent challenge.
Also Read:
- Assessing Emotional Intelligence in Large Language Models: Introducing MME-Emotion
- RelayFormer: Scalable Manipulation Localization for Visual Content
This underscores AEGIS’s unique complexity and realism, pushing the boundaries beyond the current generalization capabilities of existing models. The paper concludes that AEGIS is an indispensable evaluation benchmark, fundamentally advancing research towards developing genuinely robust, reliable, and broadly generalizable video authenticity detection methodologies capable of addressing real-world forgery threats. For more details, you can refer to the full research paper: AEGIS: Authenticity Evaluation Benchmark for AI-Generated Video Sequences.
