Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models

2026-04-01

AI Summary

This work addresses the limitations of existing vision-language models (VLMs) in autonomous driving evaluation, which has been confined to open-loop, static question-answering tasks that fail to capture closed-loop performance under out-of-distribution states and error accumulation. To bridge this gap, we propose VLM4AD, the first closed-loop benchmark for evaluating VLMs in autonomous driving. Built on the CARLA simulation platform, VLM4AD leverages DriveCommenter to dynamically generate behavior-grounded question-answer pairs, particularly covering extreme off-trajectory scenarios. The framework integrates multimodal inputs, graph-structured chain-of-thought reasoning, and a configurable control module, enabling VLMs to be directly embedded into closed-loop systems. This design facilitates fair comparisons with conventional autonomous agents in complex driving environments. We release all code and annotated datasets, establishing the first comprehensive evaluation of VLMs in closed-loop autonomous driving.

Abstract

With the rise of vision-language models (VLM), their application for autonomous driving (VLM4AD) has gained significant attention. Meanwhile, in autonomous driving, closed-loop evaluation has become widely recognized as a more reliable validation method than open-loop evaluation, as it can evaluate the performance of the model under cumulative errors and out-of-distribution inputs. However, existing VLM4AD benchmarks evaluate the model`s scene understanding ability under open-loop, i.e., via static question-answer (QA) dataset. This kind of evaluation fails to assess the VLMs performance under out-of-distribution states rarely appeared in the human collected datasets.To this end, we present Bench2Drive-VL, an extension of Bench2Drive that brings closed-loop evaluation to VLM-based driving, which introduces: (1) DriveCommenter, a closed-loop generator that automatically generates diverse, behavior-grounded question-answer pairs for all driving situations in CARLA,including severe off-route and off-road deviations previously unassessable in simulation. (2) A unified protocol and interface that allows modern VLMs to be directly plugged into the Bench2Drive closed-loop environment to compare with traditional agents. (3) A flexible reasoning and control framework, supporting multi-format visual inputs and configurable graph-based chain-of-thought execution. (4) A complete development ecosystem. Together, these components form a comprehensive closed-loop benchmark for VLM4AD. All codes and annotated datasets are open sourced.

Problem

• vision-language models
• autonomous driving
• closed-loop evaluation
• out-of-distribution
• benchmark

Innovation

• closed-loop evaluation
• vision-language models
• autonomous driving
• behavior-grounded QA
• simulation benchmark