A Comparison of Fusion Techniques for Multi-Modal Human Activity Recognition on the HARMES Dataset

2026-06-26

AI Summary

This study addresses the lack of systematic comparisons among sensor fusion methods under a unified multimodal human activity recognition benchmark. For the first time, it conducts a head-to-head evaluation of seven mainstream fusion strategies—including gated fusion and late concatenation—on the HARMES dataset, leveraging IMU, audio, and humidity signals to recognize 15 categories of daily activities. The experiments employ deep learning architectures with leave-one-participant-out cross-validation. Results demonstrate that gated multimodal fusion achieves the best performance, attaining a macro F1-score of 0.82, which represents a 6-percentage-point improvement over baseline methods, thereby confirming its effectiveness and superiority in multimodal activity recognition.

Abstract

Recent advances in Human Activity Recognition (HAR) from wearable sensors have shown that multi-modal deep learning models consistently outperform their uni-modal counterparts. Modalities can include IMUs, RGB cameras, audio signals, and others. One important aspect of multi-modal deep learning is the sensor fusion approach we apply. Over recent years, multiple fusion paradigms have been proposed for multi-modal HAR. However, to the best of our knowledge, no head-to-head comparison of these paradigms exists on a common multi-modal HAR benchmark dataset. To address this research gap, we systematically compare seven state-of-the-art sensor fusion methods on the recently released HARMES dataset, which comprises 61 hours of fully labeled IMU, audio, and ambient humidity data. The chosen dataset focuses on 15 household and personal hygiene activities of daily living (ADLs). By applying the seven different fusion techniques to a state-of-the-art multi-modal model architecture, we show that Gated Multi-modal Fusion achieves the highest macro F1-score (0.82), surpassing the concatenation-based late fusion HARMES paper baseline of 0.76 by +6pp under leave-one-participant-out evaluation. All code used in our experiments is made publicly available on GitHub.

Problem

• multi-modal fusion
• human activity recognition
• sensor fusion
• HARMES dataset
• fusion techniques

Innovation

• multi-modal fusion
• human activity recognition
• HARMES dataset
• gated fusion
• sensor fusion comparison