Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studies

2025-05-29

AI Summary

This study addresses the high cost, long duration, and limited sample size inherent in user research during early prototyping. To overcome these challenges, we propose Agentic H-CI—a framework leveraging 240 LLM-based anthropomorphic agent players to automate participant screening, interactive experience delivery, and feedback analysis. Each agent incurs only $0.28 in cost and completes evaluation in 6.9 minutes, enabling scalable (hundreds of participants), medium-fidelity UX assessment. We present the first empirical validation demonstrating that agent players achieve Pareto-optimality in behavioral consistency (82.5% persona alignment) and design insight validity. By integrating controllable personality generation, contextual background modeling, and content-analysis-driven insight extraction, the framework yields 11 actionable user insights and 6 concrete design recommendations. Developer evaluations confirm its optimal trade-off across fidelity, cost, timeliness, and insight value.

Technology Category

• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Multiagent Systems: Mechanism Design

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Agentic search

Abstract

We demonstrate the potential of anthropomorphized language agents to generate budget-friendly, moderate-fidelity, yet sufficiently insightful user experiences at scale, supporting fast, early-stage prototyping. We explore this through the case of prototyping Large Language Model-driven non-player characters (NPCs). We present Agentic H-CI, a framework that mirrors traditional user research processes-surveying, screening, experiencing, and collecting feedback and insights-with simulated agents. Using this approach, we easily construct a team of 240 player agents with a balanced range of player types and personality traits, at extremely low cost ($0.28/player) and minimal time commitment (6.9 minutes/player). Content analysis shows that agent-based players behave in ways aligned with their simulated backgrounds, achieving 82.5% alignment with designated profiles. From their interactions, we distill 11 user insights and 6 design implications to guide further development. To evaluate practical value, we conduct parallel user studies with human participants recruited locally and via crowdsourcing. Ratings from three professional game developers show that the agentic player team offers a Pareto-optimal and well-balanced trade-off across fidelity, cost, time efficiency, and insight helpfulness.

Problem

• Scaling user studies with low-cost, moderate-fidelity agents
• Simulating diverse player types for early-stage prototyping
• Balancing trade-offs between cost, time, and insight quality

Innovation

• Anthropomorphized language agents for scalable UX
• Agentic H-CI framework mirrors user research processes
• Low-cost simulated agents achieve high profile alignment