EdgeWisePersona: A Dataset for On-Device User Profiling from Natural Language Interactions

2025-05-16

AI Summary

This work addresses the challenge of inaccurate user profile reconstruction by small language models (SLMs) on edge devices. We propose a lightweight user behavior modeling framework tailored for multi-turn natural language interaction in smart homes. Methodologically, we introduce structured “routine behaviors” as semantic anchors for user profile representation; design an LLM-driven, controllable dialogue synthesis paradigm to generate high-fidelity, context-aware interaction data; and construct the first on-device benchmark and dataset for user behavior reconstruction. Experimental results demonstrate that current SLMs significantly underperform large language models (LLMs) in routine behavior reconstruction—validating the feasibility of on-device modeling while revealing a critical performance gap. This work establishes a reproducible evaluation benchmark, a systematic methodology, and concrete optimization directions for privacy-preserving, low-latency, lightweight behavior AI.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

This paper introduces a novel dataset and evaluation benchmark designed to assess and improve small language models deployable on edge devices, with a focus on user profiling from multi-session natural language interactions in smart home environments. At the core of the dataset are structured user profiles, each defined by a set of routines - context-triggered, repeatable patterns of behavior that govern how users interact with their home systems. Using these profiles as input, a large language model (LLM) generates corresponding interaction sessions that simulate realistic, diverse, and context-aware dialogues between users and their devices. The primary task supported by this dataset is profile reconstruction: inferring user routines and preferences solely from interactions history. To assess how well current models can perform this task under realistic conditions, we benchmarked several state-of-the-art compact language models and compared their performance against large foundation models. Our results show that while small models demonstrate some capability in reconstructing profiles, they still fall significantly short of large models in accurately capturing user behavior. This performance gap poses a major challenge - particularly because on-device processing offers critical advantages, such as preserving user privacy, minimizing latency, and enabling personalized experiences without reliance on the cloud. By providing a realistic, structured testbed for developing and evaluating behavioral modeling under these constraints, our dataset represents a key step toward enabling intelligent, privacy-respecting AI systems that learn and adapt directly on user-owned devices.

Problem

• Assessing small language models for on-device user profiling
• Reconstructing user profiles from natural language interactions
• Bridging performance gap between small and large language models

Innovation

• Dataset for on-device user profiling
• LLM generates realistic interaction sessions
• Benchmarks compact vs large language models