The Impostor is Among Us: Can Large Language Models Capture the Complexity of Human Personas?

2025-01-08

AI Summary

This study investigates the credibility and limitations of large language model (LLM)-generated user personas in human-centered design, specifically examining their capacity to represent human multifacetedness and diversity. To address this, we systematically compare ten pairs of expert-crafted and LLM-generated personas using a standardized HCI persona framework—guiding both prompt engineering and evaluation criteria—and integrate empirical user surveys with qualitative analysis across three dimensions: authenticity, informativeness, and diversity. Results indicate that while LLM-generated personas exhibit higher consistency and information density, they significantly underperform human-authored counterparts in contextual depth and individual differentiation, and tend to reinforce societal stereotypes. Critically, this work introduces the novel methodology of “diversity calibration”—a systematic approach to mitigate bias and enhance fairness and inclusivity in LLM-based user modeling—thereby establishing a foundational pathway for ethically robust persona generation in HCI practice.

Technology Category

• Humans and AI: User Experience and Usability
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large Language Models (LLMs) created new opportunities for generating personas, which are expected to streamline and accelerate the human-centered design process. Yet, AI-generated personas may not accurately represent actual user experiences, as they can miss contextual and emotional insights critical to understanding real users' needs and behaviors. This paper examines the differences in how users perceive personas created by LLMs compared to those crafted by humans regarding their credibility for design. We gathered ten human-crafted personas developed by HCI experts according to relevant attributes established in related work. Then, we systematically generated ten personas and compared them with human-crafted ones in a survey. The results showed that participants differentiated between human-created and AI-generated personas, with the latter being perceived as more informative and consistent. However, participants noted that the AI-generated personas tended to follow stereotypes, highlighting the need for a greater emphasis on diversity when utilizing LLMs for persona creation.

Problem

• Large Language Models
• Character Generation
• Human Diversity

Innovation

• Large Language Models
• Character Diversity
• Human Multifacetedness