Personality Without Persons? A Psychometric Critique of Big Five Testing in Large Language Models

2026-07-02

AI Summary

This study addresses the widespread but unvalidated practice of directly applying human Big Five personality inventories to characterize large language models (LLMs). It presents the first systematic evaluation of these instruments’ psychometric suitability for LLMs, assessing content validity through established psychometric criteria and conducting large-scale empirical analyses across 244 models from 49 model families. The findings reveal that standard human personality scales exhibit insufficient content validity for LLMs, with inter-model variance in Big Five scores extremely low (only 3%). Moreover, the expected five-factor structure collapses into a single dominant factor, and alignment training significantly induces socially desirable responding. These results demonstrate that existing personality inventories are ill-suited for non-human agents, offering a critical methodological warning for personality assessment and governance of LLMs.

Abstract

Human personality inventories are increasingly used to characterize large language models (LLMs), compare systems, and inform downstream governance claims. Yet, these inventories were developed and validated for humans, and it remains unclear whether they apply to LLMs. We present a systematic psychometric evaluation of Big Five personality measurements in LLMs. We ask three research questions: Do Big Five inventories a) appropriately describe LLMs, b) capture inter-individual differences across models, and c) reflect internal factors consistent with human personality. We assess content validity of five candidate Big Five inventories and administer the winning inventory to N = 244 different models spanning 49 model families. First, we found that Big Five items adapted for LLMs can reach sufficient content validity, while original human-developed items did not. Second, Big Five inventories did not capture meaningful differences between LLMs: We found low variability between models, accounting for only 3% of total score variance. Third, LLMs responses did not recover the Big Five five-factor structure with four of the Big Five facets collapsing into one (r >= .92). Direct comparisons between base and instruction-tuned model variants suggested that alignment training systematically shifted Big Five scores toward socially desirable traits. These findings demonstrate that Big Five scores do not measure a construct equivalent to human personality in LLMs. Applying human personality frameworks to LLMs produces misleading characterizations used to benchmark, compare, and govern LLMs. We highlight the need for evaluation frameworks that are developed for LLMs, rather than adopting human constructs without validation.

Problem

• personality assessment
• Big Five
• large language models
• psychometric validity
• human-AI comparison

Innovation

• psychometric evaluation
• Big Five personality
• large language models
• content validity
• alignment training