Sleepless Nights, Sugary Days: Creating Synthetic Users with Health Conditions for Realistic Coaching Agent Interactions

2025-02-18

AI Summary

Existing evaluation of health-coaching dialogue agents suffers from a lack of high-fidelity synthetic users. Method: We propose an end-to-end framework that jointly models clinical-grade health conditions (e.g., insomnia, diabetes) with real-world behavioral data to generate structured health behavior representations and fine-grained user profiles; subsequently, we simulate realistic interactions via the Concordia generative agent architecture and LLM-driven prompting. Contribution/Results: This work introduces the first joint modeling of clinical health states and multidimensional behavioral features, significantly enhancing synthetic users’ fidelity in need comprehension and challenge articulation. In two case studies, coaching agents achieved a 37% improvement in user-need understanding accuracy. Multi-expert blind evaluation rated synthetic user fidelity at 92% relative to real users—substantially outperforming all baselines.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Intelligent User Interfaces
• Multiagent Systems: Modeling other Agents

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

We present an end-to-end framework for generating synthetic users for evaluating interactive agents designed to encourage positive behavior changes, such as in health and lifestyle coaching. The synthetic users are grounded in health and lifestyle conditions, specifically sleep and diabetes management in this study, to ensure realistic interactions with the health coaching agent. Synthetic users are created in two stages: first, structured data are generated grounded in real-world health and lifestyle factors in addition to basic demographics and behavioral attributes; second, full profiles of the synthetic users are developed conditioned on the structured data. Interactions between synthetic users and the coaching agent are simulated using generative agent-based models such as Concordia, or directly by prompting a language model. Using two independently-developed agents for sleep and diabetes coaching as case studies, the validity of this framework is demonstrated by analyzing the coaching agent's understanding of the synthetic users' needs and challenges. Finally, through multiple blinded evaluations of user-coach interactions by human experts, we demonstrate that our synthetic users with health and behavioral attributes more accurately portray real human users with the same attributes, compared to generic synthetic users not grounded in such attributes. The proposed framework lays the foundation for efficient development of conversational agents through extensive, realistic, and grounded simulated interactions.

Problem

• Generates synthetic users for health coaching evaluations
• Simulates interactions using generative agent-based models
• Validates framework with expert-evaluated user-coach interactions

Innovation

• Generates synthetic users realistically
• Uses agent-based models for simulation
• Validates with expert-blind evaluations