TLDR: FE2E is a new AI model that excels at predicting 3D geometry (like depth and surface normals) from a single image. It achieves this by adapting powerful image editing models, rather than image generation models, for the task. The model introduces specific training improvements, including a “consistent velocity” objective and logarithmic data handling, allowing it to achieve state-of-the-art results with significantly less training data than previous methods.
In the rapidly evolving field of artificial intelligence, predicting the 3D geometry of a scene from a single 2D image is a crucial task for applications ranging from augmented reality to 3D reconstruction. This involves estimating properties like depth (how far objects are) and surface normals (the orientation of surfaces). Traditionally, researchers have explored leveraging powerful text-to-image (T2I) generative models, like Stable Diffusion, for these ‘dense prediction’ tasks.
However, a recent research paper titled “FROM EDITOR TO DENSE GEOMETRY ESTIMATOR” by Jiyuan Wang, Chunyu Lin, Lei Sun, Rongying Liu, Lang Nie, Mingxing Li, Kang Liao, Xiangxiang Chu, and Yao Zhao, proposes a new and highly effective approach. The authors argue that image editing models, which are inherently designed for image-to-image transformations, are a more suitable foundation for fine-tuning for dense geometry estimation than T2I generative models.
The Core Idea: Editors Over Generators
The paper introduces a novel framework called FE2E (From Editor to Estimator). The central insight is that image editing models possess inherent structural priors and a deeper understanding of input images. This allows them to converge more stably and achieve higher performance when adapted for dense geometry prediction. In contrast, generative models, initially built for creating images from text, require significant reshaping of their internal features to learn geometric cues, a process that is less efficient and stable.
FE2E’s Innovative Approach
FE2E is built upon an advanced editing model based on the Diffusion Transformer (DiT) architecture, specifically Step1X-Edit. To tailor this editor for the deterministic nature of geometry prediction, the researchers introduced several key innovations:
Consistent Velocity Training Objective: Image editing models often use a ‘flow matching loss’ that can lead to curved trajectories during inference, introducing errors for high-precision tasks like depth estimation. FE2E reformulates this by training the model to predict a ‘consistent velocity’ from a fixed starting point. This ensures a straight integration path, eliminating approximation errors and significantly reducing inference time while improving accuracy.
Logarithmic Quantization: Modern editing models typically operate with BF16 (BFloat16) precision, which is sufficient for RGB image outputs. However, dense geometry tasks demand much higher numerical precision. Directly using BF16 for depth values (e.g., 0-80 meters) would lead to unacceptable errors, especially at close ranges. FE2E addresses this by employing logarithmic quantization, which transforms depth values into a logarithmic scale before processing. This method maintains good precision across both near and far ranges, making the BF16 format viable without sacrificing accuracy or increasing computational cost.
Cost-Free Joint Estimation: Depth and surface normals are intrinsically linked, both contributing to a unified 3D representation. Leveraging the global attention mechanism of the DiT architecture, FE2E can jointly estimate both depth and normals in a single forward pass without any additional computational cost. This design allows the supervisory signals from both tasks to mutually enhance each other, leading to improved overall performance.
Also Read:
- Advancing 3D Vision with Geometric Deep Learning for Enhanced Perception and Reconstruction
- Advancing 3D Vision in Endoscopy with Adaptive Depth Estimation
Impressive Performance Gains
The results are compelling. FE2E achieves state-of-the-art performance in zero-shot monocular depth and normal estimation across multiple datasets. Notably, it demonstrates over 35% performance gains on the ETH3D dataset and outperforms the DepthAnything series, which is trained on 100 times more data. This highlights the effectiveness of harnessing the inherent capabilities of editing models rather than simply scaling up training data.
The research paper can be accessed for more details: FROM EDITOR TO DENSE GEOMETRY ESTIMATOR.
In conclusion, FE2E represents a significant step forward in dense geometry prediction, demonstrating that image editing models offer a more stable, effective, and data-efficient foundation for these complex tasks. By bridging the gap between image editing and geometry estimation, this framework opens new avenues for future research in computer vision.
