VOLA: Improving Open-World Driving by VLM-Based Semantic Attribute Prediction

2026-08-12

AI Summary

This work addresses the challenge of open-world autonomous driving by moving beyond category-based recognition to directly assess drivability and collision risk posed by unseen objects—such as debris or animals. The authors propose reframing scene understanding as dense, action-oriented attribute prediction, assigning each pixel a 7-level drivability score and a 5-level vulnerability label. Their method uniquely leverages image tokens from Qwen-VL as spatial semantic representations, combined with a lightweight boundary-aware decoder, enabling attribute transfer to unknown obstacles without requiring text generation or external mask models. Evaluated on a novel dense-attribute dataset built in CARLA, the model significantly outperforms baselines on real-world anomalous objects, achieving a vulnerability recall of 69.4%—a relative improvement of 12.3% over the best purely visual approach and 15.5% over prompt-based vision-language models.

Abstract

Driving in the real world is open-world: a car may encounter a fallen mattress, a deer, or other objects outside its training data. Naming them is not enough. The system must know how to treat each region: can it drive over it, and how severe would a collision be? We therefore shift scene perception from category labels to dense action-relevant attributes, where each pixel is labeled by how it should affect motion rather than by object name. We instantiate this general formulation with two ordered attributes: 7-rank drivability and 5-rank vulnerability. We read Qwen3.5 image-token hidden states directly as a spatial semantic representation. A lightweight boundary-aware decoder then turns this coarse token grid into sharp full-resolution attribute maps. The whole process requires neither autoregressive text generation nor an external mask model such as SAM. We train on dense attribute labels built in CARLA and test transfer to real scenes and to novel obstacles never seen in training. We compare with vision-only segmenters trained on the same attributes and prompted VLM segmenters. Our model matches strong vision-only segmenters on familiar categories and improves transfer to real open-world anomalies, reaching 69.4% mean vulnerability-rank recall versus 57.1% for the best vision-only baseline and 53.9% for the best prompted VLM baseline. These results show that VLM image tokens provide useful semantic cues for transferring driving attributes to objects outside the training vocabulary.

Problem

• open-world driving
• semantic attributes
• drivability
• vulnerability
• out-of-distribution objects

Innovation

• VLM-based semantic attribute prediction
• open-world driving
• dense action-relevant attributes
• boundary-aware decoding
• zero-shot transfer