Design and Evaluation of Generative Agent-based Platform for Human-Assistant Interaction Research: A Tale of 10 User Studies

2025-05-15

AI Summary

Human-computer interaction (HCI) research has long been constrained by the high cost, privacy risks, and scalability limitations of human-subject experiments. To address this, we introduce the first large language model (LLM)-based generative agent simulation platform for assistant-style AI interaction modeling. Our approach employs a multi-agent architecture, role-aware behavioral modeling, social reasoning simulation, and an end-to-end framework for autonomous interaction trajectory generation and evaluation. Crucially, we conduct the first systematic validation of generative agents across ten real-world user studies, achieving 82–91% alignment with empirical findings—substantially surpassing rule-based simulators in fidelity and reducing reliance on manual intervention. The platform significantly lowers experimental costs and ethical risks while enabling scalable, reproducible HCI research. We publicly release the platform and all synthetic interaction data, establishing a new paradigm for large-scale, ethically sustainable evaluation of AI assistants.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Human-Computer Interaction
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Designing and evaluating personalized and proactive assistant agents remains challenging due to the time, cost, and ethical concerns associated with human-in-the-loop experimentation. Existing Human-Computer Interaction (HCI) methods often require extensive physical setup and human participation, which introduces privacy concerns and limits scalability. Simulated environments offer a partial solution but are typically constrained by rule-based scenarios and still depend heavily on human input to guide interactions and interpret results. Recent advances in large language models (LLMs) have introduced the possibility of generative agents that can simulate realistic human behavior, reasoning, and social dynamics. However, their effectiveness in modeling human-assistant interactions remains largely unexplored. To address this gap, we present a generative agent-based simulation platform designed to simulate human-assistant interactions. We identify ten prior studies on assistant agents that span different aspects of interaction design and replicate these studies using our simulation platform. Our results show that fully simulated experiments using generative agents can approximate key aspects of human-assistant interactions. Based on these simulations, we are able to replicate the core conclusions of the original studies. Our work provides a scalable and cost-effective approach for studying assistant agent design without requiring live human subjects. We will open source both the platform and collected results from the experiments on our website: https://dash-gidea.github.io/.

Problem

• Challenges in designing personalized proactive assistant agents
• Limitations of existing HCI methods and simulated environments
• Unexplored effectiveness of LLMs in human-assistant interactions

Innovation

• Generative agent-based simulation platform for interactions
• Replicates human studies using LLM-simulated agents
• Scalable cost-effective assistant design without humans