VIMCAN: Visual-Inertial 3D Human Pose Estimation with Hybrid Mamba-Cross-Attention Network

2026-05-08

AI Summary

Existing Transformer-based multimodal 3D human pose estimation methods struggle with long sequences due to quadratic computational complexity and inadequately model spatial dependencies when fusing visual and inertial data. This work proposes VIMCAN, the first approach to integrate the Mamba state space model with cross-attention mechanisms: the former efficiently captures long-range temporal dynamics, while the latter precisely models cross-modal spatial dependencies, enabling robust fusion of RGB-derived keypoints and IMU measurements. The method achieves state-of-the-art performance with MPJPEs of 17.2 mm on TotalCapture and 45.3 mm on 3DPW, outperforming existing approaches, and enables real-time inference (>60 FPS) on consumer-grade hardware.

Abstract

The rapid advances in deep learning have significantly enhanced the accuracy of multimodal 3D human pose estimation (HPE). However, the state-of-the-art (SOTA) HPE pipelines still rely on Transformers, whose quadratic complexity makes real-time processing for long sequences impractical. Mamba addresses this issue through selective state-space modeling, enabling efficient sequence processing without sacrificing representational power. Nevertheless, it struggles to capture complex spatial dependencies in multimodal settings. To bridge this gap, we propose VIMCAN, a hybrid architecture that combines the efficient sequence modeling of Mamba with the spatial reasoning of Cross-Attention, and performs robust visual-inertial fusion and human pose estimation between RGB keypoints and wearable IMU data. By leveraging Mamba's dynamic parameterization for temporal modeling and Attention for spatial dependency extraction, VIMCAN achieves superior accuracy, with mean per-joint position errors (MPJPE) of 17.2 mm on TotalCapture and 45.3 mm on 3DPW. VIMCAN outperforms prior Transformer-based and other SOTA approaches while supporting real-time inference at over 60 frames per second on consumer-grade hardware. The source code is available on GitHub.

Problem

• 3D human pose estimation
• visual-inertial fusion
• real-time processing
• spatial dependencies
• multimodal learning

Innovation

• Mamba
• Cross-Attention
• Visual-Inertial Fusion
• 3D Human Pose Estimation
• Efficient Sequence Modeling