MPDrive: Improving Spatial Understanding with Marker-Based Prompt Learning for Autonomous Driving

2025-04-01

AI Summary

In AD-VQA, existing methods encode spatial coordinates as textual tokens, exacerbating the vision–language semantic gap and impairing spatial reasoning. To address this, we propose MPDrive—a novel framework introducing tokenized spatial representation: coordinates are mapped to semantically consistent, learnable visual tokens, enabling end-to-end spatial encoding. We further design a dual-granularity visual prompting mechanism—integrating scene-level and instance-level cues—comprising detection-expert-guided tokenized image generation, joint feature fusion of original and tokenized images, and detection-prior-enhanced instance feature extraction; these prompts are injected into large language models. Evaluated on DriveLM and CODA-LM, MPDrive achieves state-of-the-art performance, with particularly pronounced gains (+4.2% absolute accuracy) on complex spatial reasoning tasks, demonstrating superior geometric understanding and cross-modal alignment.

Technology Category

• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Autonomous driving visual question answering (AD-VQA) aims to answer questions related to perception, prediction, and planning based on given driving scene images, heavily relying on the model's spatial understanding capabilities. Prior works typically express spatial information through textual representations of coordinates, resulting in semantic gaps between visual coordinate representations and textual descriptions. This oversight hinders the accurate transmission of spatial information and increases the expressive burden. To address this, we propose a novel Marker-based Prompt learning framework (MPDrive), which represents spatial coordinates by concise visual markers, ensuring linguistic expressive consistency and enhancing the accuracy of both visual perception and spatial expression in AD-VQA. Specifically, we create marker images by employing a detection expert to overlay object regions with numerical labels, converting complex textual coordinate generation into straightforward text-based visual marker predictions. Moreover, we fuse original and marker images as scene-level features and integrate them with detection priors to derive instance-level features. By combining these features, we construct dual-granularity visual prompts that stimulate the LLM's spatial perception capabilities. Extensive experiments on the DriveLM and CODA-LM datasets show that MPDrive achieves state-of-the-art performance, particularly in cases requiring sophisticated spatial understanding.

Problem

• Bridges semantic gaps between visual coordinates and textual descriptions
• Enhances spatial understanding in autonomous driving visual QA
• Improves accuracy of visual perception and spatial expression

Innovation

• Marker-based visual prompts for spatial representation
• Dual-granularity features combining scene and instance levels
• Detection expert converts coordinates to visual markers