Helpful assistant or fruitful facilitator? Investigating how personas affect language model behavior

2024-07-02

arXiv.org

AI Summary

This study investigates how persona assignments (e.g., teacher, woman, LGBTQ+ individual) systematically influence the behavioral outputs of large language models (LLMs). Method: We conduct a large-scale empirical analysis across 12 categories comprising 162 distinct personas, evaluating responses from seven mainstream LLMs on five task domains—mathematical reasoning, historical knowledge, value alignment, and other objective/subjective benchmarks—while rigorously controlling for confounding factors via 30 prompt rewriting techniques. Contribution/Results: We provide the first evidence that persona-induced behavioral variation substantially exceeds standard prompt sensitivity; certain effects—including enhanced logical consistency under authority-related personae and richer value expression under diverse identity personae—exhibit cross-model robustness. Statistical significance is confirmed across all models and datasets, establishing persona as a potent, controllable intervention factor. These findings introduce a novel paradigm for behavior-aware model design and controllable content generation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

One way to personalize and steer generations from large language models (LLM) is to assign a persona: a role that describes how the user expects the LLM to behave (e.g., a helpful assistant, a teacher, a woman). This paper investigates how personas affect diverse aspects of model behavior. We assign to seven LLMs 162 personas from 12 categories spanning variables like gender, sexual orientation, and occupation. We prompt them to answer questions from five datasets covering objective (e.g., questions about math and history) and subjective tasks (e.g., questions about beliefs and values). We also compare persona's generations to two baseline settings: a control persona setting with 30 paraphrases of"a helpful assistant"to control for models' prompt sensitivity, and an empty persona setting where no persona is assigned. We find that for all models and datasets, personas show greater variability than the control setting and that some measures of persona behavior generalize across models.

Problem

• Investigates how personas affect language model behavior
• Examines impact of 162 personas across 12 categories on LLMs
• Compares persona-driven outputs to control and empty settings

Innovation

• Assigning diverse personas to steer LLM behavior
• Comparing persona effects across multiple datasets
• Measuring persona variability against control settings