SensorLM: Learning the Language of Wearable Sensors

2025-06-10

AI Summary

Aligning wearable sensor signals with natural language remains challenging due to semantic heterogeneity and the scarcity of large-scale, high-quality paired sensor–language annotations. Method: We introduce the first multimodal foundation model for wearable sensing. (1) We design a hierarchical captioning pipeline to construct the largest real-world sensor–language dataset to date—59.7 million hours of sensor data from over 100,000 individuals, each paired with descriptive natural language captions; (2) We generalize existing contrastive and generative multimodal architectures (e.g., CLIP, CoCa) into a unified, scalable cross-modal alignment pretraining framework. Contribution/Results: Our model achieves state-of-the-art performance on human activity recognition and healthcare tasks, enabling zero-shot classification, few-shot transfer, and cross-modal retrieval. It demonstrates label efficiency, strong generalization to unseen tasks, and robustness under limited supervision—establishing a new foundation for language-grounded wearable sensing.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present SensorLM, a family of sensor-language foundation models that enable wearable sensor data understanding with natural language. Despite its pervasive nature, aligning and interpreting sensor data with language remains challenging due to the lack of paired, richly annotated sensor-text descriptions in uncurated, real-world wearable data. We introduce a hierarchical caption generation pipeline designed to capture statistical, structural, and semantic information from sensor data. This approach enabled the curation of the largest sensor-language dataset to date, comprising over 59.7 million hours of data from more than 103,000 people. Furthermore, SensorLM extends prominent multimodal pretraining architectures (e.g., CLIP, CoCa) and recovers them as specific variants within a generic architecture. Extensive experiments on real-world tasks in human activity analysis and healthcare verify the superior performance of SensorLM over state-of-the-art in zero-shot recognition, few-shot learning, and cross-modal retrieval. SensorLM also demonstrates intriguing capabilities including scaling behaviors, label efficiency, sensor captioning, and zero-shot generalization to unseen tasks.

Problem

• Aligning sensor data with language lacks annotated datasets
• Generating hierarchical captions for sensor data understanding
• Improving recognition and retrieval in wearable sensor tasks

Innovation

• Hierarchical caption generation for sensor data
• Multimodal pretraining architectures extension
• Largest sensor-language dataset curation