HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarios

2025-06-06

AI Summary

To address the trade-off between generalization and real-time performance in long-tail driving scenarios, this paper proposes a dual-branch end-to-end autonomous driving framework: a “fast” branch for low-latency control output and a “slow” branch leveraging multi-view images and 4-second ego-motion history to invoke a vision-language model (VLM) for selective five-view prompting and multi-stage chain-of-reasoning—comprising scene understanding, decision-making, and trajectory inference—to generate high-level driving intent. A spline-based trajectory post-processing module further enhances action smoothness. The method is supervised on the Waymo Open Dataset and achieved second place in the 2025 Waymo Vision-Only End-to-End Challenge, attaining a Rater Feedback Score of 7.7367—2.77% higher than the baseline—demonstrating significantly improved generalization to long-tail scenarios and enhanced decision rationality.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We present HaoMo Vision-Language Model (HMVLM), an end-to-end driving framework that implements the slow branch of a cognitively inspired fast-slow architecture. A fast controller outputs low-level steering, throttle, and brake commands, while a slow planner-a large vision-language model-generates high-level intents such as"yield to pedestrian"or"merge after the truck"without compromising latency. HMVLM introduces three upgrades: (1) selective five-view prompting with an embedded 4s history of ego kinematics, (2) multi-stage chain-of-thought (CoT) prompting that enforces a Scene Understanding ->Driving Decision ->Trajectory Inference reasoning flow, and (3) spline-based trajectory post-processing that removes late-stage jitter and sharp turns. Trained on the Waymo Open Dataset, these upgrades enable HMVLM to achieve a Rater Feedback Score (RFS) of 7.7367, securing 2nd place in the 2025 Waymo Vision-based End-to-End (E2E) Driving Challenge and surpassing the public baseline by 2.77%.

Problem

• Enhance driving decisions in long-tailed scenarios using vision-language model
• Improve trajectory inference with multi-stage reasoning and post-processing
• Achieve low-latency high-level intent generation for autonomous driving

Innovation

• Selective five-view prompting with kinematics history
• Multi-stage chain-of-thought reasoning flow
• Spline-based trajectory post-processing for smoothness