OmniSpace: Efficient Geometry Awareness for Autonomous Vehicles MLLMs

2026-06-21

AI Summary

This work addresses the complexity and cascading failure issues in multimodal large language models (MLLMs) for autonomous driving, which arise from reliance on 3D modules during spatial reasoning. The authors propose a plug-and-play, end-to-end trainable paradigm that operates solely on 2D inputs. By integrating camera pose injection, multi-view epipolar attention, and 3D geometric knowledge distillation into MLLM training—novel contributions to the field—the model achieves significantly enhanced cross-view correspondence and depth estimation without requiring any additional 3D components at inference time. This approach demonstrates strong geometric awareness and consistently outperforms existing methods across multiple benchmarks, including nuScenes, Bench2Drive planning, nuInstruct risk detection, Omnidrive language tasks, and DriveBench generalization tests.

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable performance on 2D visual tasks, yet enhancing their spatial intelligence for real-world applications such as Autonomous Vehicles (AV) remains an open challenge. Existing geometry-aware MLLMs typically rely on auxiliary 3D models at inference time, introducing pipeline complexity and the risk of cascading failures. In this paper, we present OmniSpace, a simple yet effective plug-and-play paradigm for geometry-aware spatial reasoning from purely 2D observations. Motivated by our finding that current MLLMs are bottlenecked by weak cross-view correspondence and depth estimation, OmniSpace introduces a Camera Pose Injector, a Multi-view Epipolar Attention module, and a 3D Geometric Distillation objective that jointly address these two limitations by transferring geometric knowledge into the model. Extensive experiments show that OmniSpace surpasses existing methods on planning benchmarks (nuScenes, Bench2Drive), risk detection (nuInstruct), language (Omnidrive), and generalization (DriveBench).

Problem

• spatial intelligence
• Autonomous Vehicles
• geometry awareness
• Multimodal Large Language Models
• 3D reasoning

Innovation

• Geometry-aware MLLMs
• Camera Pose Injector
• Multi-view Epipolar Attention
• 3D Geometric Distillation
• Autonomous Vehicles