Geometry-Grounded Unified 3D Perception for Autonomous Driving

2026-08-13

AI Summary

This work addresses the limitation of existing image-based autonomous driving perception methods, which lack explicit metric geometric structure and struggle to maintain consistent 3D scene representations across multiple cameras. To overcome this, the authors propose GeoUP, a unified 3D perception framework that explicitly preserves metric geometry by integrating reconstruction-oriented latent representations, multi-view and temporal attention mechanisms, and calibration-aware raymap encoding. GeoUP models cross-image interactions through decomposed self-attention, view attention, and temporal attention, enabling joint multi-task training for depth estimation, 3D object detection, and semantic occupancy prediction. Extensive experiments demonstrate that GeoUP achieves state-of-the-art performance across multiple benchmarks—including nuScenes, Argoverse 2, and Waymo—validating the effectiveness and generalization capability of its geometric representation.

Abstract

Camera-based autonomous driving perception requires a shared representation that preserves metric 3D structure across synchronized multi-camera streams. However, existing image-based frameworks often rely on backbones pretrained for semantic recognition, and introduce 3D geometry through downstream task-specific modules. As a result, their shared representations may fail to preserve explicit metric geometry and consistent 3D scene structure. In this paper, we present a Geometry-grounded Unified 3D Perception (GeoUP) framework that adapts the reconstruction-oriented latent of VGGT to calibrated, streaming multi-camera driving scenes. GeoUP factorizes cross-image interaction into self, temporal, and view attention to capture structurally distinct temporal and cross-view correspondences. It further injects calibration-aware raymap encodings to provide metric scale and camera geometry. The resulting geometry-grounded latent is decoded for metric depth estimation, 3D object detection, and semantic occupancy prediction, corresponding to surface-, instance-, and volume-level readouts of the same 3D scene. Through joint multi-task and multi-dataset training, GeoUP effectively leverages heterogeneous annotations and generalizes across diverse sensor configurations and perception ranges. Extensive experiments on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD demonstrate that GeoUP achieves SOTA performance across detection, occupancy, and depth estimation. These results validate the effectiveness of geometry-grounded representations for unified 3D driving perception.

Problem

• 3D perception
• metric geometry
• multi-camera
• unified representation
• autonomous driving

Innovation

• geometry-grounded representation
• unified 3D perception
• multi-camera fusion
• raymap encoding
• multi-task learning