MuRAL: A Multi-Resident Ambient Sensor Dataset Annotated with Natural Language for Activities of Daily Living

2025-04-29

AI Summary

Existing HAR datasets (e.g., CASAS, ARAS) lack the semantic richness, contextual completeness, and fine-grained natural language annotations required for large language models (LLMs), hindering explainable, socially aware activity understanding. To address this, we introduce the first LLM-oriented multimodal sensor dataset for multi-occupant residential environments: it spans 21 hours of real-world home activity, synchronously capturing PIR, door-contact, and other sensor modalities, annotated with occupant identities, high-level activity labels, and fine-grained natural language descriptions. We propose a prompt-engineering-based evaluation framework comprising three benchmark tasks—agent attribution, action description, and activity classification. Experiments demonstrate LLMs’ zero-shot semantic understanding capability while exposing critical limitations in multi-user ambiguity resolution and sensor-contextual inference. The dataset is publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Recent advances in Large Language Models (LLMs) have shown promising potential for human activity recognition (HAR) using ambient sensors, especially through natural language reasoning and zero-shot learning. However, existing datasets such as CASAS, ARAS, and MARBLE were not originally designed with LLMs in mind and therefore lack the contextual richness, complexity, and annotation granularity required to fully exploit LLM capabilities. In this paper, we introduce MuRAL, the first Multi-Resident Ambient sensor dataset with natural Language, comprising over 21 hours of multi-user sensor data collected from 21 sessions in a smart-home environment. MuRAL is annotated with fine-grained natural language descriptions, resident identities, and high-level activity labels, all situated in dynamic, realistic multi-resident settings. We benchmark MuRAL using state-of-the-art LLMs for three core tasks: subject assignment, action description, and activity classification. Our results demonstrate that while LLMs can provide rich semantic interpretations of ambient data, current models still face challenges in handling multi-user ambiguity and under-specified sensor contexts. We release MuRAL to support future research on LLM-powered, explainable, and socially aware activity understanding in smart environments. For access to the dataset, please reach out to us via the provided contact information. A direct link for dataset retrieval will be made available at this location in due course.

Problem

• Lack of contextual richness in existing HAR datasets for LLMs
• Need for fine-grained natural language annotations in multi-resident settings
• Challenges in handling multi-user ambiguity with current LLMs

Innovation

• First multi-resident ambient sensor dataset with natural language
• Fine-grained natural language annotations for activity recognition
• Benchmarked with state-of-the-art LLMs for core tasks