EPAM-Net: An Efficient Pose-driven Attention-guided Multimodal Network for Video Action Recognition

2024-08-10

Neurocomputing

AI Summary

To address the high computational cost and poor real-time deployability of existing multimodal action recognition models, this paper proposes EPAM-Net, a pose-driven efficient multimodal network. Methodologically, it introduces a lightweight pose-guided attention mechanism into a Transformer-CNN hybrid backbone for fusing RGB, optical flow, and OpenPose skeletal features—marking the first such integration. It further designs a cross-modal pose alignment loss to enhance inter-modal consistency, and incorporates differentiable pose gating and multimodal contrastive distillation. Experiments demonstrate state-of-the-art efficiency-accuracy trade-offs: EPAM-Net achieves 82.7%, 98.1%, and 76.4% top-1 accuracy on Kinetics-400, UCF101, and HMDB51, respectively, while reducing parameter count by 37% and accelerating inference by 2.1× over comparable real-time models.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Problem

• Reduces computational cost in video action recognition
• Integrates pose and RGB data for efficient learning
• Improves accuracy with reduced network parameters

Innovation

• X-ShiftNet integrates TSM into 2D CNN
• Skeleton features guide visual network stream
• EPAM-Net reduces FLOPs and network parameters