A Pedestrian-Vehicle Interaction Benchmark and Annotation Framework for Unstructured Scenes via Uncalibrated Cameras

2026-05-25

AI Summary

Existing public datasets lack support for dense pedestrian–vehicle interactions in unstructured environments, hindering trajectory prediction research for autonomous driving in complex mixed-traffic scenarios. To address this gap, this work proposes a scalable annotation framework based on uncalibrated surveillance videos and introduces PINNS, the first large-scale pedestrian–vehicle interaction dataset encompassing diverse countries, scenes, and weather conditions. PINNS provides trajectories and scene-level annotations compliant with the standards of the Chinese Association of Automation and integrates key technical components, including uncalibrated video processing, multi-object trajectory extraction, cross-regional standardized annotation, and modeling of heterogeneous traffic behaviors. The release of PINNS is expected to significantly advance research on human–vehicle interaction modeling and autonomous driving in unstructured environments.

Abstract

Predicting the interaction between pedestrian and vehicle is essential for autonomous driving safety in unstructured and semi-structured scenarios; however, this task is severely hindered by the scarcity of public datasets that feature dense pedestrian-vehicle interactions. Most current studies rely on structured road data, leaving the complex, heterogeneous interactions found in unstructured environments insufficiently represented and researched. In this paper, we propose a dataset annotation framework based on video data from uncalibrated surveillance cameras and present PINNS (Pedestrian-vehicle Interaction dataset from uNcalibrated cameras in uNstructured Scenes). The dataset covers multiple countries and regions, includes diverse typical traffic scenarios, and considers variations in seasons, lighting conditions, and weather. It focuses on complex scenes with dense pedestrian-vehicle interactions and is designed to be easily extensible. The dataset is constructed and annotated according to the standard issued by the Chinese Association of Automation, providing both trajectory data and corresponding scene-level information. Furthermore, this paper analyzes current challenges and research directions in heterogeneous agent trajectory prediction, shows the necessity and usefulness of the proposed dataset. We hope our framework and dataset will facilitate research on trajectory prediction and autonomous driving in complex mixed traffic scenarios. PINNS is publicly available at https://github.com/Songan-Lab.

Problem

• pedestrian-vehicle interaction
• unstructured scenes
• trajectory prediction
• autonomous driving
• heterogeneous agents

Innovation

• uncalibrated cameras
• unstructured scenes
• pedestrian-vehicle interaction
• trajectory prediction
• heterogeneous agents