CMD-HAR: Cross-Modal Disentanglement for Wearable Human Activity Recognition

2025-03-27

AI Summary

To address challenges in wearable-sensor-based human activity recognition—including multimodal data entanglement, high activity heterogeneity, and difficulties in edge deployment—this paper proposes a cross-modal spatiotemporal disentangled representation framework coupled with gradient modulation. The method employs a modality decomposition–alignment–fusion strategy, integrating spatiotemporal attention with cross-modal disentangled representation learning to achieve feature disentanglement, enhanced generalization, and computational efficiency. A gradient modulation mechanism is further introduced to optimize multi-task joint training. Additionally, a lightweight edge-deployment simulation system is developed. Extensive experiments on multiple mainstream public datasets demonstrate that the proposed approach improves recognition accuracy by an average of +2.3%, reduces model parameters by 37%, and decreases inference latency by 41%, thereby validating its effectiveness, robustness, and practicality in real-world edge scenarios.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Human Activity Recognition (HAR) is a fundamental technology for numerous human - centered intelligent applications. Although deep learning methods have been utilized to accelerate feature extraction, issues such as multimodal data mixing, activity heterogeneity, and complex model deployment remain largely unresolved. The aim of this paper is to address issues such as multimodal data mixing, activity heterogeneity, and complex model deployment in sensor-based human activity recognition. We propose a spatiotemporal attention modal decomposition alignment fusion strategy to tackle the problem of the mixed distribution of sensor data. Key discriminative features of activities are captured through cross-modal spatio-temporal disentangled representation, and gradient modulation is combined to alleviate data heterogeneity. In addition, a wearable deployment simulation system is constructed. We conducted experiments on a large number of public datasets, demonstrating the effectiveness of the model.

Problem

• Address multimodal data mixing in wearable activity recognition
• Resolve activity heterogeneity via disentangled representation
• Simplify complex model deployment for wearable systems

Innovation

• Spatiotemporal attention modal decomposition alignment fusion
• Cross-modal spatio-temporal disentangled representation
• Wearable deployment simulation system