Adaptive Cross-Modal Fusion with Sparse Attention for Pedestrian Crossing Intention Prediction

2026-07-13

AI Summary

This work addresses the limitations of existing pedestrian crossing intention prediction methods, which often rely on single-modality inputs or dense multimodal fusion, leading to inefficient integration of visual and motion cues and redundant information. To overcome these challenges, the authors propose the ADAPT framework, which employs a sparse cross-modal attention mechanism to selectively model critical modality interactions while jointly capturing local and global visual context alongside temporal motion dynamics. The architecture integrates weight-shared Swin Transformer V2 blocks, cross-modal guided attention, Mamba-based motion encoding, and ViT-based temporal fusion. Evaluated on the JAAD and PIE benchmarks, ADAPT achieves state-of-the-art performance with an AUC of 0.85 on JAAD and an accuracy of 0.92 on PIE, while maintaining real-time efficiency through a per-sample inference latency of only 17.23 ms.

Abstract

Predicting pedestrian crossing intention is a safety-critical task for autonomous driving, yet existing approaches often rely on single-modal inputs or dense multimodal fusion strategies that inadequately capture complementary visual and kinematic information while introducing redundant inter-modal interactions. We propose ADAPT (Adaptive Domain-Aware Pedestrian Crossing Transformer), a multimodal framework that jointly models local and global visual context together with temporal motion dynamics for accurate pedestrian crossing intention prediction. ADAPT processes four spatially aligned visual modalities, including RGB images, local depth maps, global semantic maps, and global depth maps, together with ego-vehicle speed, pedestrian bounding boxes, and skeleton pose information through five specialized modules: a weight-shared Swin Transformer V2 backbone for visual feature extraction, a Cross-Modality Guided Attention module for hierarchical visual fusion, a Mamba-based Motion Feature Encoding module for efficient temporal modeling, a Sparse Cross-Modal Attention module that selectively preserves the most informative inter-modal interactions, and a Vision Transformer-based Temporal Feature Fusion module for sequence-level prediction. Extensive experiments on the JAAD and PIE benchmark datasets demonstrate that ADAPT consistently outperforms existing state-of-the-art methods while maintaining low computational complexity. On JAAD, the proposed method achieves an AUC of 0.73 on JAADbeh and 0.85 on JAADall, while on PIE it achieves an accuracy of 0.92 and an AUC of 0.90. Furthermore, ADAPT performs inference in only 17.23 ms per sample, offering an effective balance between predictive accuracy and real-time deployment efficiency for intelligent transportation and autonomous driving applications.

Problem

• pedestrian crossing intention prediction
• multimodal fusion
• cross-modal interaction
• visual-kinematic complementarity
• redundant inter-modal interactions

Innovation

• Sparse Cross-Modal Attention
• Multimodal Fusion
• Mamba-based Temporal Modeling
• Adaptive Pedestrian Intention Prediction
• Efficient Real-time Inference