In Shift and In Variance: Assessing the Robustness of HAR Deep Learning Models Against Variability

2025-01-01

Italian National Conference on Sensors

AI Summary

This work addresses the robustness evaluation of deep learning models for human activity recognition (HAR) in real-world scenarios, where performance degradation arises from multiple sources of distributional variability—including inter-subject differences, sensor device heterogeneity, and variations in wearable placement and orientation. Method: Leveraging the HARVAR and REALDISP benchmark datasets, we conduct a systematic empirical analysis of how these factors jointly impact model generalization. We quantify distribution shifts using the Maximum Mean Discrepancy (MMD) metric and correlate them with observed accuracy drops. Contribution/Results: We provide the first quantitative evidence that MMD exhibits a strong inverse correlation with test accuracy—confirming its validity as an interpretable, task-agnostic robustness diagnostic. Crucially, we demonstrate that even a single source of variability induces significant performance degradation, while compound variations lead to severe generalization failure. Our findings establish MMD-based distribution shift estimation as a principled, pre-deployment assessment framework for HAR model robustness and generalizability.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Deep learning (DL)-based Human Activity Recognition (HAR) using wearable inertial measurement unit (IMU) sensors can revolutionize continuous health monitoring and early disease prediction. However, most DL HAR models are untested in their robustness to real-world variability, as they are trained on limited lab-controlled data. In this study, we isolated and analyzed the effects of the subject, device, position, and orientation variabilities on DL HAR models using the HARVAR and REALDISP datasets. The Maximum Mean Discrepancy (MMD) was used to quantify shifts in the data distribution caused by these variabilities, and the relationship between the distribution shifts and model performance was drawn. Our HARVAR results show that different types of variability significantly degraded the DL model performance, with an inverse relationship between the data distribution shifts and performance. The compounding effect of multiple variabilities studied using REALDISP further underscores the challenges of generalizing DL HAR models to real-world conditions. Analyzing these impacts highlights the need for more robust models that generalize effectively to real-world settings. The MMD proved valuable for explaining the performance drops, emphasizing its utility in evaluating distribution shifts in HAR data.

Problem

• Assessing robustness of HAR models to real-world variabilities.
• Evaluating impact of variability on data distribution shifts.
• Developing robust DL HAR models for real-world applications.

Innovation

• Assessed DL HAR models using HARVAR and REALDISP datasets.
• Used Maximum Mean Discrepancy to measure data shifts.
• Analyzed variability impact on model performance robustness.