Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Prediction

2026-06-22

AI Summary

This work addresses the challenge of jointly modeling temporal dynamics, multimodal cues, and social interactions in multi-pedestrian trajectory prediction by proposing a three-stage hierarchical Transformer architecture that sequentially performs temporal encoding, multimodal fusion, and scene-level interaction reasoning. By explicitly decoupling these three modeling tasks, the method introduces a lightweight GRU-based summarization mechanism and a time-agent social attention module, achieving a balance between computational efficiency and interpretability. Evaluated on real-world datasets including JRDB and Urban, the approach achieves state-of-the-art performance and demonstrates strong results on JTA, particularly excelling at forecasting complex interactive behaviors such as early turning maneuvers.

Abstract

Pedestrian trajectory prediction requires modeling temporal dynamics, multimodal cues, and social interactions in crowded environments. Existing methods often address these factors separately or entangle them in costly attention blocks, limiting scalability, flexibility, and interpretability. We propose a three-step hierarchical Transformer that explicitly separates temporal encoding, multimodal fusion, and scene-level interaction reasoning. Lightweight GRU summaries enable efficient cross-modal attention, while social attention over time--agent tokens captures inter-pedestrian influences at manageable cost. Experiments on JTA, JRDB, and the Pedestrians and Cyclists in Road Traffic dataset show state-of-the-art performance on real-world datasets (JRDB, Urban) and competitive results on JTA. Ablation and qualitative analyses confirm the contribution of each stage and the model's ability to anticipate complex behaviors such as early turning.

Problem

• pedestrian trajectory prediction
• temporal dynamics
• multimodal cues
• social interactions
• crowded environments

Innovation

• Hierarchical Transformer
• Multi-pedestrian Trajectory Prediction
• Multimodal Fusion
• Social Interaction Modeling
• Lightweight Attention