Can LLMs Truly Embody Human Personality? Analyzing AI and Human Behavior Alignment in Dispute Resolution

2026-02-07

AI Summary

This study investigates whether large language models (LLMs) can authentically replicate human behavioral patterns in conflict resolution as shaped by personality traits. To this end, we propose the first interpretable evaluation framework for aligning AI behavior with human behavior, grounded in the Big Five personality model. We construct a dialogue dataset that pairs specific personality profiles with conflict scenarios and employ quantifiable behavioral metrics to compare LLMs’ strategic choices and conflict outcomes against those of humans. Experimental results reveal significant discrepancies between current mainstream LLMs and human behavior in personality-driven conflict interactions, raising critical concerns about the reliability of these models as behavioral proxies in social applications.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) are increasingly used to simulate human behavior in social settings such as legal mediation, negotiation, and dispute resolution. However, it remains unclear whether these simulations reproduce the personality-behavior patterns observed in humans. Human personality, for instance, shapes how individuals navigate social interactions, including strategic choices and behaviors in emotionally charged interactions. This raises the question: Can LLMs, when prompted with personality traits, reproduce personality-driven differences in human conflict behavior? To explore this, we introduce an evaluation framework that enables direct comparison of human-human and LLM-LLM behaviors in dispute resolution dialogues with respect to Big Five Inventory (BFI) personality traits. This framework provides a set of interpretable metrics related to strategic behavior and conflict outcomes. We additionally contribute a novel dataset creation methodology for LLM dispute resolution dialogues with matched scenarios and personality traits with respect to human conversations. Finally, we demonstrate the use of our evaluation framework with three contemporary closed-source LLMs and show significant divergences in how personality manifests in conflict across different LLMs compared to human data, challenging the assumption that personality-prompted agents can serve as reliable behavioral proxies in socially impactful applications. Our work highlights the need for psychological grounding and validation in AI simulations before real-world use.

Problem

• personality-behavior alignment
• large language models
• dispute resolution
• Big Five personality
• human-AI behavioral simulation

Innovation

• personality-behavior alignment
• dispute resolution
• large language models
• evaluation framework
• Big Five Inventory