Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions

2026-07-01

AI Summary

This study investigates the instability of persona-driven generation (PDG) in large language models when applied to multiple-choice question answering (MCQA), with a particular focus on the lack of persona consistency in non-free-text outputs. To address this, we introduce the first multidimensional stability evaluation framework encompassing performance, output, and question-level correctness, enabling a systematic analysis of how model scale, domain, prompt format, and hyperparameters influence stability. Our findings reveal that prompt format exerts a stronger effect on instability than hyperparameters such as temperature; mathematical and commonsense questions are especially prone to instability; and the best- or worst-performing personas vary significantly across configurations. These results underscore the necessity of explicitly evaluating hyperparameter-induced stability in PDG applications and demonstrate its close relationship with task accuracy.

Abstract

Persona-driven generations (PDGs) have seen prolific use in research and industry applications, where a large language model (LLM) takes on a 'persona' while completing some task. While persona expressed through free-form text (like dialogue) has substantial work investigating stability or consistency, relatively, persona expressed in non-text-heavy outputs (like in multiple-choice question answering, or MCQA) is often overlooked. We work to address this gap, seeking to understand the instability of LLM PDGs in MCQA tasks. We develop three metrics investigating the performance, outcome, and question correctness stability, evaluating three distinct dimensions. Using these metrics, we find that instability varies consistently between model families and model size, and across question domains, with math/commonsense questions leading to greater instability. We also find task prompt format introduces more prediction instability than other hyperparameters, like temperature. Finally, we find that instability is related to task accuracy, and using our instability metrics, find different experimental settings that result in different best and worst personas for tasks, despite their similarity. This reveals the importance of checking hyperparameter instability in PDGs.

Problem

• persona-driven generation
• LLM instability
• multiple-choice question answering
• hyperparameter sensitivity
• model consistency

Innovation

• Persona-Driven Generation
• Instability Metrics
• Multiple-Choice Question Answering
• Prompt Format Sensitivity
• Large Language Models