Human Simulacra: Benchmarking the Personification of Large Language Models

2024-02-28

AI Summary

This study investigates the feasibility and efficacy of large language models (LLMs) as human surrogates in social science research, specifically for anthropomorphic personality modeling and cognitive process simulation. Method: We integrate cognitive modeling, prompt engineering, and structured behavioral protocols to develop three core innovations: (1) the first end-to-end framework for virtual agent lifecycle construction, enabling coherent life-history generation from scratch; (2) a multi-agent–based cognitive simulation mechanism that explicitly models internal thought streams and decision logic; and (3) a psychology-informed dual-perspective evaluation paradigm—self-report and observer-based—that jointly assesses subjective consistency and external behavioral validity. Results: Experiments demonstrate significant improvements in response alignment with target personality profiles. All code and datasets are publicly released, establishing a reproducible, empirically verifiable paradigm for LLM-driven social experimentation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) are recognized as systems that closely mimic aspects of human intelligence. This capability has attracted attention from the social science community, who see the potential in leveraging LLMs to replace human participants in experiments, thereby reducing research costs and complexity. In this paper, we introduce a framework for large language models personification, including a strategy for constructing virtual characters' life stories from the ground up, a Multi-Agent Cognitive Mechanism capable of simulating human cognitive processes, and a psychology-guided evaluation method to assess human simulations from both self and observational perspectives. Experimental results demonstrate that our constructed simulacra can produce personified responses that align with their target characters. Our work is a preliminary exploration which offers great potential in practical applications. All the code and datasets will be released, with the hope of inspiring further investigations. Our code and dataset are available at: https://github.com/hasakiXie123/Human-Simulacra.

Problem

• Develop framework for personifying large language models.
• Simulate human cognitive processes using multi-agent mechanisms.
• Evaluate human-like responses from self and observational perspectives.

Innovation

• Framework for LLM personification with virtual characters
• Multi-Agent Cognitive Mechanism simulating human cognition
• Psychology-guided evaluation for human-like simulation assessment