Reinforcing Human Behavior Simulation via Verbal Feedback

2026-05-19

AI Summary

Current large language models struggle to effectively leverage subjective, multidimensional verbal feedback when simulating human behavior, resulting in limited behavioral fidelity. This work addresses this gap by introducing verbal feedback as a first-class signal in reinforcement learning and proposes a novel training paradigm that generalizes without requiring feedback during testing. The authors establish SOUL, a unified benchmark spanning six task categories, and develop the GRPO algorithm to jointly optimize both the original model outputs and their feedback-conditioned refinements, thereby enabling knowledge distillation from verbal instructions into the base policy. The resulting DITTO model achieves an average performance gain of 36% on SOUL and outperforms GPT-5.4 on six out of ten tasks, substantially enhancing the realism of human behavior simulation.

Abstract

Humans learn social norms and behaviors from verbal feedback (e.g., a parent saying "that was rude" or a friend explaining "here's why that hurt"). Yet, learning from feedback for LLMs has largely focused on domains like code and math, where RL rewards are directly verifiable and condensed into scalar values. As LLMs are increasingly used to simulate human behavior, e.g., standing in for users, patients, students, and other personas, there is a pressing need to make them more human-like, which requires embracing a fundamentally different kind of signal: feedback that is verbal, subjective, and multi-faceted. We present DITTO, a model trained by treating verbal feedback as a first-class signal in reinforcement learning. After each rollout, DITTO receives verbal feedback and generates a feedback-conditioned improved rollout; both outputs are jointly optimized with GRPO, distilling verbal guidance into the base policy without requiring feedback at test time. We also introduce SOUL (Simulation gym Of hUman-Like behavior), a unified benchmark and training data suite spanning 10 tasks across six categories: Theory of Mind, character role play, social skill, learner simulation, user simulation, and persona simulation. DITTO achieves an average 36% improvement over the base model and exceeds GPT-5.4 on 6 of 10 SOUL benchmarks, demonstrating that RL with verbal feedback is a promising direction for training LLMs to simulate human behavior.

Problem

• verbal feedback
• human behavior simulation
• reinforcement learning
• large language models
• subjective feedback

Innovation

• verbal feedback
• reinforcement learning
• human behavior simulation
• GRPO
• SOUL benchmark