DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models

2026-06-15

AI Summary

Existing methods for evaluating autonomous driving struggle to balance interpretability with contextual awareness: rule-based metrics lack situational understanding, while vision-language models (VLMs) produce ambiguous outputs with insufficient physical grounding. This work proposes DriveJudge, the first framework that integrates VLM-based contextual reasoning with physically grounded, deterministic rule functions to enable interpretable and context-aware assessment. The contributions include a large-scale, annotated dataset of challenging driving scenarios, two human-aligned benchmark tasks, and state-of-the-art performance—achieving a 21.23% higher AUC than EPDMS on driving quality classification and outperforming DriveCritic by 6.5% on trajectory preference selection—thereby establishing a new standard for accurate and interpretable evaluation in autonomous driving.

Abstract

Autonomous driving has shifted towards end-to-end policy learning, where reliable, interpretable policy evaluation is a fundamental challenge as driving quality is highly context-dependent. Commonly used rule-based driving metrics like EPDMS are interpretable but lack context-awareness, while recent VLMbased evaluations are context-aware but limited by ambiguous VLM outputs and weak physical grounding. To evaluate driving in a manner that is both interpretable and context-aware, we introduce DriveJudge. DriveJudge is a driving evaluation agent that combines rule-grounded evaluation with Vision-Language Model (VLM) reasoning and selectively invokes physically-grounded deterministic rule functions after interpreting the environmental context. To train and evaluate DriveJudge, we curate a large-scale dataset of 33,577 challenging driving samples with human annotations on whether the driving behavior is reasonable in the given scenario. With this dataset, we address the underexplored problem of driving metric evaluation, and introduce two human-aligned benchmark tasks: Driving Quality Classification and Trajectory Preference Selection. DriveJudge outperforms EPDMS for driving quality classification by 21.23 AUC, and the recent VLM-based DriveCritic for trajectory preference selection by 6.5%, setting a new standard for interpretable and precise driving evaluation.

Problem

• autonomous driving evaluation
• context-awareness
• interpretability
• driving quality assessment
• vision-language models

Innovation

• Vision-Language Models
• Rule-Grounded Evaluation
• Context-Aware Driving Assessment
• Physically-Grounded Reasoning
• Autonomous Driving Evaluation