Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?

2026-05-16

AI Summary

This study investigates whether large language models possess stable, authentic personality traits or merely mimic superficial cues. The authors conduct supervised fine-tuning (SFT), DPO, and ORPO on models using long texts annotated with Big Five personality labels, then systematically evaluate personality expression consistency and accuracy via the IPIP-NEO questionnaire. Results show that post-training significantly reduces response variance across prompts, enhancing personality score stability. However, the accuracy of reconstructing full personality profiles remains near chance levels, suggesting that unguided text lacks sufficient personality-relevant signals for high-fidelity recovery. These findings highlight fundamental limitations in current large models’ capacity for genuine personality modeling and provide an empirical foundation for future research on controllable personality generation.

Abstract

Can large language models reliably express a human-like personality, or are they merely mimicking surface cues without a stable underlying profile? To investigate this, we induce personality in LLMs by fine-tuning them on the long-form essays, where each essay is associated with a target Big Five personality profile. We then evaluate the stability and fidelity of the induced personality using the IPIP-NEO questionnaire. Specifically, we ask: (i) does post-training (SFT, DPO, ORPO) stabilize questionnaire scores under prompt rephrasings, and (ii) can it induce target Big Five profiles from unguided essays? Our results demonstrate that fine-tuning consistently reduces variance in questionnaire responses across five models, directly mitigating the evaluation fragility reported in pre-trained models. However, this newfound stability reveals a more fundamental limitation: accuracy on the full five-dimensional profile remains near chance, even when single-trait scores improve. This indicates that unguided essays lack the cues needed for faithful personality expression. We therefore argue for scenario-grounded datasets or interactive elicitation that accumulates test-aligned evidence over time.

Problem

• personality induction
• evaluation drift
• Big Five personality
• large language models
• IPIP-NEO

Innovation

• personality induction
• evaluation drift
• large language models
• Big Five personality
• fine-tuning stability