Among Them: A game-based framework for assessing persuasion capabilities of LLMs

2025-02-27

AI Summary

This study systematically evaluates the deception and persuasion capabilities of large language models (LLMs) in controlled settings. To address the lack of standardized, theory-grounded assessment frameworks, we propose the first game-based evaluation paradigm inspired by the social deduction mechanics of *Among Us*, integrating principles from social psychology and rhetoric. We introduce a fine-grained annotation scheme covering 25 distinct persuasion strategies and design a cross-model adversarial protocol coupled with behavior-log-driven strategy detection and attribution. Empirical validation across eight state-of-the-art LLMs reveals that model scale does not correlate positively with persuasion success; conversely, longer outputs negatively impact win rates. On average, models successfully deploy 22 out of 25 strategies. We publicly release both the evaluation toolkit and benchmark dataset, establishing a reproducible, interpretable, and multidimensional quantitative framework for assessing LLMs’ societal influence capabilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

The proliferation of large language models (LLMs) and autonomous AI agents has raised concerns about their potential for automated persuasion and social influence. While existing research has explored isolated instances of LLM-based manipulation, systematic evaluations of persuasion capabilities across different models remain limited. In this paper, we present an Among Us-inspired game framework for assessing LLM deception skills in a controlled environment. The proposed framework makes it possible to compare LLM models by game statistics, as well as quantify in-game manipulation according to 25 persuasion strategies from social psychology and rhetoric. Experiments between 8 popular language models of different types and sizes demonstrate that all tested models exhibit persuasive capabilities, successfully employing 22 of the 25 anticipated techniques. We also find that larger models do not provide any persuasion advantage over smaller models and that longer model outputs are negatively correlated with the number of games won. Our study provides insights into the deception capabilities of LLMs, as well as tools and data for fostering future research on the topic.

Problem

• Assessing LLM persuasion capabilities systematically
• Comparing LLM models using game-based framework
• Quantifying in-game manipulation with persuasion strategies

Innovation

• Game-based framework for LLM persuasion assessment
• Quantifies manipulation using 25 persuasion strategies
• Compares models via game statistics and strategy usage