X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras

2026-07-14

AI Summary

This work addresses the challenge of real-time metric depth estimation from multi-view heterogeneous cameras—specifically fisheye and pinhole models—and proposes a lightweight feedforward model that efficiently fuses inputs from both camera types within a unified framework. The method introduces learnable calibration tokens and a Jacobian-parameterized distortion bias, leveraging cross-attention to model local projective geometric variations, thereby achieving cross-camera geometric alignment and consistent depth prediction. A global scale prediction mechanism is adopted to circumvent complex auxiliary tasks. With only 0.04B parameters, the model reduces the AbsRel error by 25.4% and decreases parameter count by 88.9% on the OmniScene-Full benchmark, while achieving 41 FPS inference speed. It demonstrates strong performance across both synthetic and real-world indoor and outdoor scenes.

Abstract

We present X-lens, a compact feed-forward model for metric depth estimation from a variable number of calibrated fisheye and pinhole views. To support real-time downstream perception, X-lens is built around a geometry-aware heterogeneous camera formulation with two key components. Learnable calibration tokens provide a coarse alignment between fisheye and pinhole projective spaces, while a Jacobian-parameterized distortion bias injected into cross-attention models local projection changes and promotes cross-camera consistency, enabling robust generalization with only 0.04B parameters and up to 41 FPS. The model predicts dense depth together with a global metric scale, avoiding auxiliary reconstruction targets that increase computation and optimization complexity. To learn such cross-camera generalization at scale and depth, X-lens is trained on multiple public datasets and OmniScene, our newly released large-scale synthetic dataset containing approximately 266K synchronized six-view frames, 1.7M individual images, and 103 indoor and outdoor scenes. Extensive experiments on both real-world and synthetic indoor and outdoor datasets demonstrate superior heterogeneous-camera metric depth accuracy, reducing AbsRel by 25.4\% on OmniScene-Full over the strongest baseline while using 88.9\% fewer parameters, with competitive performance on conventional fisheye-only and pinhole-only settings.

Problem

• metric depth estimation
• heterogeneous cameras
• real-time perception
• fisheye camera
• pinhole camera

Innovation

• metric depth estimation
• heterogeneous cameras
• learnable calibration tokens
• Jacobian-parameterized distortion
• cross-attention