WatchHAR: Real-time On-device Human Activity Recognition System for Smartwatches

2025-09-04

AI Summary

To address the need for real-time, privacy-sensitive human activity recognition (HAR) on smartwatches, this work proposes the first fully on-device, multimodal (audio + IMU) fine-grained HAR system. Methodologically, we design an end-to-end trainable, lightweight architecture that unifies sensor preprocessing and inference, enables efficient multimodal feature fusion, and is optimized for on-device execution—achieving millisecond-level latency without compromising accuracy. Our system achieves >90% accuracy across 25+ fine-grained activity classes, with only 9.3 ms per event detection and 11.8 ms per multimodal classification—significantly outperforming existing state-of-the-art models. By eliminating cloud dependency, our approach mitigates privacy leakage and communication latency, enabling secure, low-latency, and scalable real-time sensing directly on wearable devices.

Technology Category

• Machine Learning: Hardware-aware ML
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

Despite advances in practical and multimodal fine-grained Human Activity Recognition (HAR), a system that runs entirely on smartwatches in unconstrained environments remains elusive. We present WatchHAR, an audio and inertial-based HAR system that operates fully on smartwatches, addressing privacy and latency issues associated with external data processing. By optimizing each component of the pipeline, WatchHAR achieves compounding performance gains. We introduce a novel architecture that unifies sensor data preprocessing and inference into an end-to-end trainable module, achieving 5x faster processing while maintaining over 90% accuracy across more than 25 activity classes. WatchHAR outperforms state-of-the-art models for event detection and activity classification while running directly on the smartwatch, achieving 9.3 ms processing time for activity event detection and 11.8 ms for multimodal activity classification. This research advances on-device activity recognition, realizing smartwatches' potential as standalone, privacy-aware, and minimally-invasive continuous activity tracking devices.

Problem

• Real-time on-device human activity recognition for smartwatches
• Addressing privacy and latency issues with external processing
• Achieving accurate multimodal classification in unconstrained environments

Innovation

• Optimized end-to-end trainable sensor processing module
• Real-time on-device multimodal activity classification
• Unified audio and inertial-based HAR architecture