🤖 AI Summary
Large language models (LLMs) exhibit suboptimal action selection and limited strategic reasoning in complex social deduction games (e.g., *Werewolf*) due to biases in pretraining data and the absence of explicit reward-driven optimization. Method: We propose a novel decoupled framework wherein the LLM handles multi-round deductive reasoning and open-ended linguistic action generation, while a separate policy network—trained via Proximal Policy Optimization (PPO)—learns optimal action selection in the continuous language-action space. Contribution/Results: This is the first work to enable strategic-level reinforcement learning optimization over open linguistic action spaces. Experiments demonstrate significant gains over pure LLM baselines; human adversarial evaluation confirms robust high-level social reasoning capabilities—including credible accusation, deception, and alliance formation—achieving human-level performance.
📝 Abstract
Agents built with large language models (LLMs) have shown great potential across a wide range of domains. However, in complex decision-making tasks, pure LLM-based agents tend to exhibit intrinsic bias in their choice of actions, which is inherited from the model's training data and results in suboptimal performance. To develop strategic language agents, i.e., agents that generate flexible language actions and possess strong decision-making abilities, we propose a novel framework that powers LLM-based agents with reinforcement learning (RL). We consider Werewolf, a popular social deduction game, as a challenging testbed that emphasizes versatile communication and strategic gameplay. To mitigate the intrinsic bias in language actions, our agents use an LLM to perform deductive reasoning and generate a diverse set of action candidates. Then an RL policy trained to optimize the decision-making ability chooses an action from the candidates to play in the game. Extensive experiments show that our agents overcome the intrinsic bias and outperform existing LLM-based agents in the Werewolf game. We also conduct human-agent experiments and find that our agents achieve human-level performance and demonstrate strong strategic play.
