Step-Level Preference Learning for Generative Agents in Social Simulations

2026-07-15

AI Summary

This work addresses the limitation of existing large language model–based generative agents in authentically simulating human behavior, primarily due to the absence of human preference supervision over intermediate decision-making processes such as planning, memory retrieval, and reflection. To bridge this gap, the study introduces step-level human preferences as a training signal, leveraging an interactive simulation interface to construct a fine-grained dataset comprising 57,000 annotated trajectories. The authors optimize open-source language models through a combination of supervised fine-tuning (SFT) and direct preference optimization (DPO). Experimental results demonstrate that this approach significantly enhances the agents’ local decision rationality and long-term behavioral fidelity in social simulations, leading to improved collaboration capabilities and interaction quality.

Abstract

Large language model (LLM)-based generative agents simulate human behavior through long-horizon decision-making processes that comprise intermediate steps such as planning, memory retrieval, reflection, and action selection. However, fine-grained human annotations of these intermediate steps remain scarce, and existing agents are not grounded in human preferences over such intermediate decisions. To address this gap, we introduce \method, an interactive simulation interface that enables us to collect step-level human preference supervision over agent decision trajectories, leading to a dataset of 57K fine-grained annotations. We conduct step-level preference learning on open-weight language models using supervised finetuning and direct preference optimization on this data, consistently improving simulation fidelity, coordination, and interaction quality, and inducing more socially effective agent behavior. Our results show that step-level human supervision is an effective training signal for improving both local decision quality and long-horizon agent behavior.

Problem

• generative agents
• step-level preference
• social simulations
• human preference
• decision-making

Innovation

• step-level preference learning
• generative agents
• social simulations
• human preference supervision
• direct preference optimization