What's Hidden Matters: Identifying Planning-Critical Occluded Agents using Vision-Language Models

2026-06-30

AI Summary

Autonomous driving systems often struggle to assess the actual impact of occluded traffic participants on motion planning in complex scenarios, leading to either overly conservative behavior or misjudged risks. This work proposes Planning KL Divergence (PKL), a novel metric that leverages vision-language models (VLMs) to rank occluded objects by their planning-criticality and generate structured reasoning annotations. The study introduces the first systematic training framework and benchmark focused on high-impact occlusions, employing a PKL-guided data selection strategy. Experiments on nuScenes demonstrate that a small VLM fine-tuned with PKL-selected data significantly outperforms large zero-shot models, with PKL-based sampling yielding approximately 30% performance improvement over random sampling.

Abstract

Autonomous vehicles must safely navigate complex environments where planning-critical agents may be hidden from view. Current approaches often treat all occlusions with uniform conservatism, yielding needlessly defensive driving, or they infer hidden spaces without estimating the impact on the planner. This work bridges the critical gap between perception and planning by enabling Vision-Language Models (VLMs) to identify and reason about the specific hidden agents that are most critical to the ego-vehicle's trajectory. We introduce a novel framework that uses Planning KL-divergence (PKL), an information-theoretic metric, to systematically identify and rank occluded agents based on their impact on the ego vehicle's plan. Using this planning-aware ranking, we employ an expert VLM (GPT-5) to generate rich, structured annotations that capture the visual evidence and reasoning required for this task. We apply this framework to the nuScenes dataset to create a new benchmark focused on high-impact scenarios. We conduct comprehensive experiments on a wide range of general-purpose and domain-adapted VLMs, demonstrating that fine-tuning on our PKL-guided data yields dramatic performance improvements across all models. Notably, our results show that smaller, fine-tuned models significantly outperform their much larger zero-shot counterparts, and that our PKL-guided data selection strategy improves performance by approximately 30\% over random sampling. Our work presents the first systematic approach for training VLMs to focus on planning-critical occlusions, enabling more semantically grounded and efficient risk assessment in autonomous driving.

Problem

• occluded agents
• autonomous driving
• planning-critical
• perception-planning gap
• risk assessment

Innovation

• Vision-Language Models
• Planning KL-divergence
• Occluded Agent Reasoning
• Autonomous Driving Planning
• PKL-guided Data Selection