TLDR: PICABench is a new benchmark and dataset introduced to evaluate and improve the physical realism of AI-generated image edits. It assesses models across optics, mechanics, and state transitions, using a region-aware evaluation protocol (PICAEval) to accurately identify physics violations. The accompanying PICA-100K dataset, derived from synthetic videos, helps train models to achieve more physically consistent results, revealing that current AI models still struggle significantly with generating realistic physical effects in image editing.
Image editing technology has made incredible strides, allowing us to manipulate images with complex instructions. However, a crucial aspect often overlooked is whether these edits look physically real. For instance, if you remove an object, its shadow and reflections should also disappear, and nearby objects should react realistically. Unfortunately, current models and evaluation methods primarily focus on just completing the instruction, often neglecting these physical effects.
To address this gap, a team of researchers introduced PICABench, a new benchmark designed to systematically evaluate how physically realistic image edits are. This benchmark covers eight specific sub-dimensions across three main categories: Optics, Mechanics, and State Transition. These categories encompass common editing operations like adding, removing, or changing attributes of objects in an image.
Understanding Physical Realism in Image Editing
The PICABench framework breaks down physical realism into three intuitive dimensions:
- Optics: This dimension checks if edits follow the basic rules of light. This includes how shadows are cast, how light reflects off surfaces, how light bends through transparent materials (refraction), and the effects of light sources. For example, if you add a lamp, its light should realistically illuminate the scene, casting appropriate shadows and highlights.
- Mechanics: This category assesses whether edited objects remain mechanically and causally consistent with the scene. It looks at deformation (e.g., rigid objects shouldn’t bend like rubber, and soft objects should deform naturally under pressure) and causality (e.g., objects shouldn’t float without support, and removing a support should cause an object to fall or tilt realistically).
- State Transition: This dimension evaluates whether environmental and material changes unfold in a physically coherent manner. This can be global, like changing the time of day or season, where all visual cues (lighting, shadows, vegetation) should update consistently. It can also be local, involving specific objects, such as making a bench wet or freezing a soda can, where the changes should integrate smoothly with the surrounding context and material properties.
PICAEval: A Smarter Way to Evaluate
Evaluating physical correctness is tricky because there’s no “perfect” reference image. Traditional methods often rely on general prompts that vision-language models (VLMs) might misinterpret or “hallucinate” answers for. To overcome this, the researchers developed PICAEval, a more reliable and interpretable evaluation protocol. It uses a VLM-as-a-judge approach but with targeted, per-example questions aligned with specific physical sub-dimensions. Crucially, it incorporates human-annotated “key regions” in the image, directing the VLM’s attention to the most relevant areas for physical evidence (like reflection surfaces or contact points). This method significantly improves diagnostic accuracy and aligns better with human assessments.
Learning Physics from Videos with PICA-100K
Beyond just evaluating, the research also explores solutions. They propose PICA-100K, a large synthetic dataset of 100,000 editing examples created from videos. The idea is that videos naturally capture real-world physics and dynamics. By using advanced text-to-image models as scene renderers and image-to-video models as state-transition simulators, they can generate realistic video clips. From these videos, they extract “before” and “after” image pairs along with detailed editing instructions. Fine-tuning existing image editing models on PICA-100K has shown to significantly improve their ability to generate physically realistic edits without sacrificing the overall quality of the image.
Also Read:
- Unlocking 6D Spatial Reasoning in AI: A New Benchmark for Multimodal Models
- UrbanVerse: Creating Realistic City Simulations from Online Videos for AI Training
The Current State of Image Editing
After evaluating 11 mainstream image editing models (both open-source and closed-source), the researchers found that physical realism remains a significant challenge. Most models struggle to produce physically consistent results, often failing to account for realistic lighting, deformations, or state changes. Even unified multi-modal models, which are good at understanding visual content, don’t necessarily translate this understanding into better physical realism during generation. This highlights a persistent gap between semantic understanding and the ability to apply physical laws in image generation.
The work presented in this paper, available at https://arxiv.org/pdf/2510.17681, offers a crucial step forward. By providing a comprehensive benchmark, a reliable evaluation protocol, and a novel dataset for training, PICABench aims to serve as a foundation for future research, pushing image editing from simple content manipulation towards truly physically consistent realism.
