TLDR: A new method called “Calibration Tokens” allows existing monocular depth estimators, trained on standard perspective images, to accurately estimate depth from distorted fisheye camera images without needing to retrain the entire model. By introducing small, trainable tokens that adjust the internal representations of fisheye images to match those of perspective images, the method overcomes distortion challenges, offers high efficiency, and maintains compatibility with traditional cameras.
Monocular depth estimation, the process of determining the distance of objects from a single 2D image, is a crucial technology for applications like autonomous vehicles, robotics, and extended reality. While significant advancements have been made with Foundational Monocular Depth Estimators (FMDEs) trained on vast datasets, these models primarily learn from standard perspective images. This creates a major challenge when dealing with fisheye cameras, which are commonly used in many real-world systems due to their wider field of view.
The core problem lies in the inherent distortion of fisheye images. Unlike perspective images where straight lines remain straight, fisheye lenses cause lines to appear curved, leading to a ‘covariate shift’—a change in the input data distribution that confuses FMDEs. As a result, these powerful models produce inaccurate and blurry depth estimates when fed fisheye imagery.
Traditional approaches to address this issue have their own drawbacks. Recalibrating and undistorting images can introduce errors, latency, and visual artifacts like stretching or cropping. Alternatively, training entirely new monocular depth estimators specifically for fisheye images is difficult because large-scale fisheye datasets are scarce. Furthermore, fine-tuning existing FMDEs risks ‘parameter drift,’ where the model loses its broad generalization capabilities and becomes specialized only for fisheye cameras, complicating systems that use a mix of camera types.
A new research paper, “Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens”, proposes an innovative solution: Calibration Tokens. Instead of retraining or fine-tuning the entire FMDE, this method focuses on a lightweight adaptation mechanism. The key insight is that FMDEs are already highly capable; the issue is how fisheye images are encoded in their internal ‘latent embeddings’.
How Calibration Tokens Work
Calibration Tokens are small, trainable parameters that are appended to the input sequence of image ‘tokens’ within the Transformer-based architecture of many FMDEs. These tokens leverage the model’s attention mechanism to ‘modulate’ the latent embeddings of fisheye images, effectively aligning them with the distribution of embeddings from perspective images. This means the original FMDE remains largely unchanged, while the Calibration Tokens act as a translator, mitigating the distortion-induced covariate shift.
This design offers several significant advantages. Firstly, it’s entirely lossless for input images, as no spatial transformations are applied during inference. Secondly, the computational overhead is minimal; Calibration Tokens consist of far fewer parameters than a full vision transformer model, leading to negligible increases in memory usage and inference time (less than 1ms added latency). Thirdly, the approach is backward-compatible: simply omitting the Calibration Tokens allows the FMDE to maintain its original performance on perspective images. Finally, camera intrinsics are not required during inference, simplifying deployment by eliminating the need for arduous calibration processes.
Self-Supervised Training
To train these Calibration Tokens, the researchers developed a self-supervised objective. They exploit the abundance of publicly available perspective image datasets. During training, synthetic fisheye images are created by artificially distorting perspective images. The FMDE then infers depth maps for both the original perspective image and the synthetic fisheye image. A self-supervised loss function (LogL1) is computed by ‘undistorting’ the fisheye depth map back to the perspective reference frame, allowing for a direct comparison with the high-fidelity depth estimate from the original perspective image. This approach preserves the supervision signal and avoids introducing artifacts that would arise from transforming the perspective depth map into the fisheye frame.
Experimental Validation
The method was evaluated across various FMDEs, including MiDaS, DepthAnything, and UniDepth, on both indoor (ScanNet++) and outdoor (KITTI-360) datasets. The results consistently showed significant improvements in depth estimation accuracy for fisheye images. For instance, UniDepth with Calibration Tokens surpassed state-of-the-art methods like DepthAnyCamera and FoV A-Depth, even though Calibration Tokens were trained on a much smaller, combined indoor and outdoor dataset (200K samples) compared to methods that train separate models for each setting.
Analysis revealed that Calibration Tokens effectively modulate fisheye embeddings, making them more similar to perspective embeddings. The study also demonstrated that using ‘layer-wise’ tokens (a unique set of tokens appended at each encoder layer) is more effective than a single set of tokens. Crucially, the Calibration Token approach significantly outperformed traditional fine-tuning, which often led to a performance drop, highlighting the importance of preserving the original model’s knowledge.
Also Read:
- Enhancing 3D Scene Reconstruction with Depth and Edge Guidance
- Advancing Building Energy Renovation with Image-to-3D Facade Modeling
Future Implications
This research presents a promising general approach for adapting foundational vision models to different camera types. Its ability to extend FMDEs to fisheye images with minimal computational cost and without sacrificing backward-compatibility is a significant step forward. This could simplify the operational overhead of multi-camera systems, making it easier to integrate diverse camera inputs into a single, powerful depth estimation framework.
