A DVDrive Approach for doScenes Instructed Driving Challenge

2026-06-19

AI Summary

This study addresses the problem of autonomous driving trajectory prediction guided by natural language instructions, aiming to integrate visual scene context, historical motion dynamics, and textual commands to forecast the ego-vehicle’s trajectory over the next six seconds (12 waypoints). Building upon the OmniDrive framework, we propose a view-partitioned perception mechanism that projects query features and multi-view image tokens into localized view spaces. By incorporating visibility-aware cross-attention, our approach mitigates interference across views and enhances alignment between language instructions and local visual evidence. Evaluated on the nuScenes Instructional Driving Challenge, the proposed method achieves significant improvements in trajectory prediction accuracy. The implementation code has been made publicly available.

Abstract

Instruction-conditioned trajectory prediction is an emerging problem in autonomous driving, where a model predicts the future ego trajectory not only from visual scene context and historical motion, but also from a natural-language maneuver instruction. This paper presents our submission to the doScenes Instructed Driving Challenge, built upon OmniDrive, a vision-language-action driving agent with 3D perception, reasoning, and planning capabilities. We adapt OmniDrive to the doScenes setting by training it on instruction-annotated nuScenes scenes and generating a 6-second ego trajectory represented by 12 future waypoints. To improve multi-view visual grounding, we further introduce a DVPE-style divided-view perception module into the OmniDrive perception head. Instead of attending globally to all camera features, the proposed module groups query features and image tokens into divided local view spaces and performs visibility-aware cross-attention within each view. This design reduces irrelevant cross-view interference and helps the model better align language instructions with local driving-relevant visual evidence. The code is publicly available at: https://github.com/feel12348/doscenes-omnidrive.

Problem

• instruction-conditioned trajectory prediction
• autonomous driving
• natural-language instruction
• ego trajectory prediction
• vision-language-action

Innovation

• instruction-conditioned trajectory prediction
• divided-view perception
• visibility-aware cross-attention
• vision-language driving
• multi-view visual grounding