PEDESTRIANQA: A Benchmark for Vision-Language Models on Pedestrian Intention and Trajectory Prediction

2026-05-23

AI Summary

This work addresses the challenges of low accuracy and poor interpretability in pedestrian intention and trajectory prediction for autonomous driving by reframing the task as a visual-language question answering problem. The authors introduce PedestrianQA, a large-scale video question answering dataset, and propose the first approach leveraging large vision-language models (VLMs) to jointly reason about visual dynamics, contextual cues, and interactions among traffic participants through fine-tuning. Without relying on task-specific architectures, the method generates structured, interpretable natural language predictions. Evaluated on four benchmarks—PIE, JAAD, TITAN, and IDD-PeD—the model significantly improves intention classification accuracy, trajectory prediction precision, and explanation quality, demonstrating the potential of VLMs for unified modeling in safety-critical scenarios.

Abstract

Pedestrian intention and trajectory prediction are critical for the safe deployment of autonomous driving systems, directly influencing navigation decisions in complex traffic environments. Recent advances in large vision-language models offer a powerful new paradigm for these tasks by combining high-capacity visual understanding with flexible natural language reasoning. In this work, we introduce PedestrianQA, a large-scale video-based dataset that formulates pedestrian intention and trajectory prediction as question-answering tasks augmented with structured rationales. PedestrianQA expresses richly annotated pedestrian sequences, in natural language, enabling VLMs to learn from visual dynamics, contextual cues, and interactions among traffic agents while generating concise explanations of their predictions without needing specialized architectures tailored for each task. Empirical evaluations across PIE, JAAD, TITAN, and IDD-PeD show that finetuning state-of-the-art VLMs on PedestrianQA significantly improves intention classification, trajectory forecasting accuracy, and the quality of explanatory rationales, demonstrating the strong potential of VLMs as a unified and explainable framework for safety-critical pedestrian behavior modeling.

Problem

• pedestrian intention prediction
• trajectory prediction
• autonomous driving
• vision-language models
• safety-critical behavior modeling

Innovation

• vision-language models
• pedestrian intention prediction
• trajectory forecasting
• explainable AI
• question-answering benchmark