AgentSense: Virtual Sensor Data Generation Using LLM Agent in Simulated Home Environments

2025-06-13

AI Summary

To address the critical challenges of scarce ground-truth labeled data and poor model generalization in smart-home human activity recognition (HAR), this paper proposes an LLM-driven, persona-based virtual sensor data generation method. Within an extended VirtualHome simulation environment, it jointly models user habits, household layouts, and heterogeneous sensor configurations (e.g., infrared, door contact, power consumption) to automatically synthesize high-fidelity, behaviorally diverse, environment-aware activity sequences—without manual annotation. This work pioneers the deep integration of large language models with embodied simulation, enabling joint generation of semantically coherent behaviors and physically grounded actions. Experiments across five HAR benchmarks demonstrate that models trained on just a few days of real data augmented with synthetic data achieve performance comparable to those trained on full-scale real datasets—effectively alleviating the labeling bottleneck and substantially improving few-shot generalization.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

A major obstacle in developing robust and generalizable smart home-based Human Activity Recognition (HAR) systems is the lack of large-scale, diverse labeled datasets. Variability in home layouts, sensor configurations, and user behavior adds further complexity, as individuals follow varied routines and perform activities in distinct ways. Building HAR systems that generalize well requires training data that captures the diversity across users and environments. To address these challenges, we introduce AgentSense, a virtual data generation pipeline where diverse personas are generated by leveraging Large Language Models. These personas are used to create daily routines, which are then decomposed into low-level action sequences. Subsequently, the actions are executed in a simulated home environment called VirtualHome that we extended with virtual ambient sensors capable of recording the agents activities as they unfold. Overall, AgentSense enables the generation of rich, virtual sensor datasets that represent a wide range of users and home settings. Across five benchmark HAR datasets, we show that leveraging our virtual sensor data substantially improves performance, particularly when real data are limited. Notably, models trained on a combination of virtual data and just a few days of real data achieve performance comparable to those trained on the entire real datasets. These results demonstrate and prove the potential of virtual data to address one of the most pressing challenges in ambient sensing, which is the distinct lack of large-scale, annotated datasets without requiring any manual data collection efforts.

Problem

• Lack of large-scale labeled datasets for smart home HAR systems
• Variability in home layouts and user behavior complicates HAR
• Need for diverse training data across users and environments

Innovation

• LLM agents generate diverse virtual personas
• VirtualHome simulates activities with ambient sensors
• Combines virtual and limited real data effectively