Are Economists Always More Introverted? Analyzing Consistency in Persona-Assigned LLMs

2025-06-03

AI Summary

This work addresses the lack of persona consistency in large language models (LLMs) during role-playing—a critical gap for trustworthy agent design. We propose the first systematic, quantitative evaluation framework assessing behavioral stability across diverse tasks (e.g., questionnaire generation, multi-turn dialogue) and personas (e.g., occupational roles, personality traits, political stances). Our framework integrates multidimensional persona classification, cross-task response comparison, semantic consistency measurement, and controllable generation evaluation. Empirical analysis reveals: (1) persona consistency is non-uniformly distributed; (2) structured tasks improve consistency by up to 37%; (3) occupational personas are particularly susceptible to stereotypical biases; and (4) context augmentation and task structuring are the most effective mitigation strategies. The study uncovers synergistic effects among persona type, societal stereotypes, and model architecture, offering both theoretical insights and practical guidelines for building reliable, consistent persona-aware LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Personalized Large Language Models (LLMs) are increasingly used in diverse applications, where they are assigned a specific persona - such as a happy high school teacher - to guide their responses. While prior research has examined how well LLMs adhere to predefined personas in writing style, a comprehensive analysis of consistency across different personas and task types is lacking. In this paper, we introduce a new standardized framework to analyze consistency in persona-assigned LLMs. We define consistency as the extent to which a model maintains coherent responses when assigned the same persona across different tasks and runs. Our framework evaluates personas across four different categories (happiness, occupation, personality, and political stance) spanning multiple task dimensions (survey writing, essay generation, social media post generation, single turn, and multi-turn conversations). Our findings reveal that consistency is influenced by multiple factors, including the assigned persona, stereotypes, and model design choices. Consistency also varies across tasks, increasing with more structured tasks and additional context. All code is available on GitHub.

Problem

• Analyzing consistency in persona-assigned LLMs across tasks
• Evaluating persona adherence in writing style and response coherence
• Investigating factors affecting LLM consistency like stereotypes and design

Innovation

• Standardized framework for persona consistency analysis
• Multi-category persona evaluation across tasks
• Consistency influenced by persona and model design