Modeling 3D Pedestrian-Vehicle Interactions for Vehicle-Conditioned Pose Forecasting

2026-02-09

AI Summary

This work addresses the challenge of accurately predicting future 3D pedestrian poses in complex urban environments, where existing autonomous driving systems often fail due to the lack of explicit modeling of surrounding vehicle dynamics. To this end, we propose the first 3D pose forecasting framework that explicitly models multi-agent interactions between pedestrians and vehicles. Our approach integrates a vehicle encoder and a pedestrian–vehicle cross-attention mechanism into an extended TBIFormer architecture, conditioning pose predictions on dynamic vehicle states. We further introduce Waymo-3DSkelMo+, an enhanced dataset aligned with 3D vehicle bounding boxes, and devise a scene sampling strategy to accommodate varying interaction complexities. Experimental results demonstrate that our method significantly improves prediction accuracy, underscoring the critical role of incorporating vehicle dynamics in enhancing the safety of autonomous driving systems.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Accurately predicting pedestrian motion is crucial for safe and reliable autonomous driving in complex urban environments. In this work, we present a 3D vehicle-conditioned pedestrian pose forecasting framework that explicitly incorporates surrounding vehicle information. To support this, we enhance the Waymo-3DSkelMo dataset with aligned 3D vehicle bounding boxes, enabling realistic modeling of multi-agent pedestrian-vehicle interactions. We introduce a sampling scheme to categorize scenes by pedestrian and vehicle count, facilitating training across varying interaction complexities. Our proposed network adapts the TBIFormer architecture with a dedicated vehicle encoder and pedestrian-vehicle interaction cross-attention module to fuse pedestrian and vehicle features, allowing predictions to be conditioned on both historical pedestrian motion and surrounding vehicles. Extensive experiments demonstrate substantial improvements in forecasting accuracy and validate different approaches for modeling pedestrian-vehicle interactions, highlighting the importance of vehicle-aware 3D pose prediction for autonomous driving. Code is available at: https://github.com/GuangxunZhu/VehCondPose3D

Problem

• pedestrian pose forecasting
• vehicle-pedestrian interaction
• 3D motion prediction
• autonomous driving
• multi-agent interaction

Innovation

• vehicle-conditioned pose forecasting
• 3D pedestrian-vehicle interaction
• cross-attention fusion
• multi-agent scene modeling
• TBIFormer adaptation