Are Human Interactions Replicable by Generative Agents? A Case Study on Pronoun Usage in Hierarchical Interactions

2025-01-25

AI Summary

This study investigates whether generative language model (LLM) agents can authentically emulate human-like pronoun usage patterns in hierarchical social interactions—specifically, linguistic distinctions between leaders and non-leaders. Method: We construct a multi-agent LLM dialogue system simulating role-based social dynamics, employing quantitative pronoun frequency analysis, controlled ablation experiments, and benchmarking against authentic human corpora. Contribution/Results: We provide the first systematic evidence that current LLM agents—including those enhanced via prompt engineering and domain-specific fine-tuning—fail to spontaneously reproduce empirically established, hierarchy-sensitive pronoun patterns, even when possessing relevant world knowledge. This reveals a fundamental behavioral gap in LLMs’ capacity to internalize and enact socially situated linguistic norms. Our findings challenge prevailing assumptions about LLMs as high-fidelity social simulators and introduce a novel, quantifiable paradigm for evaluating LLMs’ sociolinguistic intelligence.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

As Large Language Models (LLMs) advance in their capabilities, researchers have increasingly employed them for social simulation. In this paper, we investigate whether interactions among LLM agents resemble those of humans. Specifically, we focus on the pronoun usage difference between leaders and non-leaders, examining whether the simulation would lead to human-like pronoun usage patterns during the LLMs' interactions. Our evaluation reveals the significant discrepancies between LLM-based simulations and human pronoun usage, with prompt-based or specialized agents failing to demonstrate human-like pronoun usage patterns. In addition, we reveal that even if LLMs understand the human pronoun usage patterns, they fail to demonstrate them in the actual interaction process. Our study highlights the limitations of social simulations based on LLM agents, urging caution in using such social simulation in practitioners' decision-making process.

Problem

• Generative Agents
• Pronoun Usage
• Hierarchical Interaction

Innovation

• Super Language Models
• Pronoun Usage
• Social Interaction Simulation