WOLF: Werewolf-based Observations for LLM Deception and Falsehoods

2025-12-09

AI Summary

Existing large language models (LLMs) exhibit limited capability in multi-agent deception detection, and mainstream evaluation paradigms reduce deception to static classification, neglecting its interactive, adversarial, and longitudinally evolving nature. Method: We introduce the first social reasoning benchmark grounded in the *Werewolf* game mechanism, proposing a novel “role-anchored + day-night cycle” dynamic interaction framework. It integrates a deception taxonomy (omission, distortion, fabrication, misdirection) with longitudinal trust modeling, enabling disentangled, fine-grained, and reproducible assessment of deception generation versus detection. Leveraging LangGraph, we implement procedural role agents with self-reported honesty, peer deception scoring, smoothed suspicion tracking, and standardized annotation. Results: Evaluated on 7,320 utterances across 100 rounds, the benchmark achieves a werewolf deception rate of 31%, peer deception detection accuracy of 71–73%, overall classification accuracy of 52%, and significantly elevated suspicion toward werewolves (>60%), effectively distinguishing genuine from deceptive roles.

Technology Category

• Multiagent Systems: Adversarial Agents
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Deception is a fundamental challenge for multi-agent reasoning: effective systems must strategically conceal information while detecting misleading behavior in others. Yet most evaluations reduce deception to static classification, ignoring the interactive, adversarial, and longitudinal nature of real deceptive dynamics. Large language models (LLMs) can deceive convincingly but remain weak at detecting deception in peers. We present WOLF, a multi-agent social deduction benchmark based on Werewolf that enables separable measurement of deception production and detection. WOLF embeds role-grounded agents (Villager, Werewolf, Seer, Doctor) in a programmable LangGraph state machine with strict night-day cycles, debate turns, and majority voting. Every statement is a distinct analysis unit, with self-assessed honesty from speakers and peer-rated deceptiveness from others. Deception is categorized via a standardized taxonomy (omission, distortion, fabrication, misdirection), while suspicion scores are longitudinally smoothed to capture both immediate judgments and evolving trust dynamics. Structured logs preserve prompts, outputs, and state transitions for full reproducibility. Across 7,320 statements and 100 runs, Werewolves produce deceptive statements in 31% of turns, while peer detection achieves 71-73% precision with ~52% overall accuracy. Precision is higher for identifying Werewolves, though false positives occur against Villagers. Suspicion toward Werewolves rises from ~52% to over 60% across rounds, while suspicion toward Villagers and the Doctor stabilizes near 44-46%. This divergence shows that extended interaction improves recall against liars without compounding errors against truthful roles. WOLF moves deception evaluation beyond static datasets, offering a dynamic, controlled testbed for measuring deceptive and detective capacity in adversarial multi-agent interaction.

Problem

• Measures deception production and detection in multi-agent systems
• Evaluates LLMs' deceptive abilities and peer deception detection
• Provides dynamic benchmark for adversarial multi-agent deception analysis

Innovation

• Multi-agent social deduction benchmark for deception measurement
• Programmable LangGraph state machine with role-grounded agents
• Standardized deception taxonomy and longitudinal suspicion tracking