Exploring Big Five Personality and AI Capability Effects in LLM-Simulated Negotiation Dialogues

2025-06-19

AI Summary

This study investigates how AI agents in high-stakes tasks adapt to human operators’ personality traits—specifically Big Five dimensions (notably Agreeableness and Extraversion)—and how agent attributes (transparency, competence, adaptability) causally influence negotiation trustworthiness, goal attainment, and empathic communication. We propose the first LLM-based agent evaluation framework integrating causal discovery with social-cognitive linguistic analysis (LIWC, MFD, Empath), validated through reproducible multi-role simulations on the Sotopia platform across two critical domains: price negotiation and human-AI job interviews. Our method quantifies causal pathways between human personality and AI behavior, identifying actionable metrics for empathic expression, moral foundations, and perspective-taking. Results demonstrate significant improvements in task effectiveness and human trust. This work establishes a novel, socially grounded reliability assessment paradigm for high-risk human-AI collaboration.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Agentic search

Abstract

This paper presents an evaluation framework for agentic AI systems in mission-critical negotiation contexts, addressing the need for AI agents that can adapt to diverse human operators and stakeholders. Using Sotopia as a simulation testbed, we present two experiments that systematically evaluated how personality traits and AI agent characteristics influence LLM-simulated social negotiation outcomes--a capability essential for a variety of applications involving cross-team coordination and civil-military interactions. Experiment 1 employs causal discovery methods to measure how personality traits impact price bargaining negotiations, through which we found that Agreeableness and Extraversion significantly affect believability, goal achievement, and knowledge acquisition outcomes. Sociocognitive lexical measures extracted from team communications detected fine-grained differences in agents'empathic communication, moral foundations, and opinion patterns, providing actionable insights for agentic AI systems that must operate reliably in high-stakes operational scenarios. Experiment 2 evaluates human-AI job negotiations by manipulating both simulated human personality and AI system characteristics, specifically transparency, competence, adaptability, demonstrating how AI agent trustworthiness impact mission effectiveness. These findings establish a repeatable evaluation methodology for experimenting with AI agent reliability across diverse operator personalities and human-agent team dynamics, directly supporting operational requirements for reliable AI systems. Our work advances the evaluation of agentic AI workflows by moving beyond standard performance metrics to incorporate social dynamics essential for mission success in complex operations.

Problem

• Evaluating AI agent adaptability in mission-critical negotiations
• Assessing personality traits' impact on negotiation outcomes
• Measuring AI trustworthiness in human-AI team dynamics

Innovation

• Uses Sotopia for LLM-simulated negotiation dialogues
• Employs causal discovery to measure personality effects
• Evaluates human-AI dynamics via transparency and adaptability