VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Driving

2026-02-24

AI Summary

Existing vision-language models (VLMs) lack the capacity for cross-view 3D geometric reasoning, limiting their effectiveness in multi-view perception and inference for autonomous driving. To address this, we propose a plug-in Cross-View Geometric Enhancement (CVGE) module that, for the first time, integrates geometric representations extracted from a frozen 3D foundation model into VLMs through a decoupled, hierarchical, and plug-and-play adaptive injection mechanism. This approach significantly enhances the geometric understanding of VLMs in complex autonomous driving scenarios without requiring any fine-tuning of the backbone model. Extensive experiments demonstrate consistent performance improvements across five autonomous driving benchmarks, spanning critical tasks such as cross-view risk perception, motion prediction, and trajectory planning.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

The significance of cross-view 3D geometric modeling capabilities for autonomous driving is self-evident, yet existing Vision-Language Models (VLMs) inherently lack this capability, resulting in their mediocre performance. While some promising approaches attempt to mitigate this by constructing Q&A data for auxiliary training, they still fail to fundamentally equip VLMs with the ability to comprehensively handle diverse evaluation protocols. We thus chart a new course, advocating for the infusion of VLMs with the cross-view geometric grounding of mature 3D foundation models, closing this critical capability gap in autonomous driving. In this spirit, we propose a novel architecture, VGGDrive, which empowers Vision-language models with cross-view Geometric Grounding for autonomous Driving. Concretely, to bridge the cross-view 3D geometric features from the frozen visual 3D model with the VLM's 2D visual features, we introduce a plug-and-play Cross-View 3D Geometric Enabler (CVGE). The CVGE decouples the base VLM architecture and effectively empowers the VLM with 3D features through a hierarchical adaptive injection mechanism. Extensive experiments show that VGGDrive enhances base VLM performance across five autonomous driving benchmarks, including tasks like cross-view risk perception, motion prediction, and trajectory planning. It's our belief that mature 3D foundation models can empower autonomous driving tasks through effective integration, and we hope our initial exploration demonstrates the potential of this paradigm to the autonomous driving community.

Problem

• Vision-Language Models
• Cross-View Geometric Grounding
• Autonomous Driving
• 3D Geometric Modeling
• VLMs

Innovation

• Cross-view geometric grounding
• Vision-Language Models
• 3D foundation models
• Autonomous driving
• Plug-and-play architecture