Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videos

2026-06-17

AI Summary

This work addresses the unrealistic "mixed-mode" predictions in pedestrian trajectory forecasting from egocentric videos, which arise due to multimodal human behaviors. To tackle this issue, the authors propose the Multimodal Mode-aware Prediction Model (MMPM), which explicitly partitions pedestrian trajectories into two semantic modes based on crossing intent. MMPM introduces a mode-aware, model-agnostic Motion Trajectory Prediction (MTP) module that can be seamlessly integrated into existing architectures to enhance performance. The framework fuses gaze, head pose, and hand gesture cues to model pedestrian–vehicle and pedestrian–environment interactions, and combines a Conditional Variational Autoencoder (CVAE) with a query-based decoder to generate mode-consistent multimodal trajectory predictions. Evaluated on the PIE and JAAD datasets, MMPM significantly reduces displacement errors and consistently improves the forecasting accuracy of diverse backbone models, including BiTrap-NP and SGNet-ED.

Abstract

Pedestrian trajectory prediction from an ego-centric camera is challenging since it depends on complex interactions with vehicles and scene context, as well as the intention of the pedestrian. By modelling correlation and intent from the historical and future trajectories of the pedestrian, it will usually result in a multimodal (i.e. multiple modes) distribution. Existing stochastic predictors often sample multiple futures from a single unimodal distribution, which can yield sub-optimal 'mixed-mode' trajectories that lie between distinct motion patterns and become implausible in real scenes. In this paper, we propose MMPM, a mode-aware framework that separately models future trajectory distributions into semantically meaningful modes based on the pedestrian's crossing behavior. MMPM consists of two modules: behavior-aware Pedestrian Interaction Module (PIM) that jointly captures pedestrian-vehicle and pedestrian-environment interactions by introducing gaze, head and hand gesture, and a CVAE-based Mode-aware Trajectory Predictor (MTP) module to model the future trajectory distributions on two modes, crossing and non-crossing the road, separately. A query-based decoder further enforces mode consistency during decoding. Experiments on PIE and JAAD datasets show that our method surpasses state-of-the-art baselines. Our proposed MTP is model-agnostic, which can be integrated into existing frameworks such as BiTrap-NP and SGNet-ED to further improve future trajectory prediction performance. We additionally introduce a data-driven validation protocol that matches predictions to spatio-temporally consistent ground-truth trajectories, demonstrating improved frame-wise displacement errors over previous work.

Problem

• pedestrian trajectory prediction
• multimodal distribution
• ego-centric videos
• crossing behavior
• mode consistency

Innovation

• multimodal trajectory prediction
• mode-aware modeling
• ego-centric video
• pedestrian behavior
• CVAE-based predictor