Large Language Models Are More Persuasive Than Incentivized Human Persuaders

2025-05-14

AI Summary

This study empirically tests whether state-of-the-art large language models (LLMs) surpass highly motivated humans in persuasive efficacy during real-time, two-way conversational interactions. Method: We conducted a large-scale, pre-registered online randomized controlled trial comparing Claude Sonnet 3.5 against monetary-incentivized human persuaders in a bidirectional dialogue task, where persuaders aimed to guide respondents toward either correct (“truth-directed”) or incorrect (“false-directed”) answers. Our design uniquely integrates behavioral economic incentive structures with LLMs’ real-time response capabilities. Contribution/Results: We provide the first experimental evidence in authentic interactive settings that LLM persuaders significantly increase respondent compliance: improving answer accuracy and participant earnings in truth-directed tasks, while markedly reducing both in false-directed tasks. These findings demonstrate that current frontier LLMs systematically outperform high-incentive human persuaders, establishing a critical empirical benchmark for assessing AI’s societal influence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We directly compare the persuasion capabilities of a frontier large language model (LLM; Claude Sonnet 3.5) against incentivized human persuaders in an interactive, real-time conversational quiz setting. In this preregistered, large-scale incentivized experiment, participants (quiz takers) completed an online quiz where persuaders (either humans or LLMs) attempted to persuade quiz takers toward correct or incorrect answers. We find that LLM persuaders achieved significantly higher compliance with their directional persuasion attempts than incentivized human persuaders, demonstrating superior persuasive capabilities in both truthful (toward correct answers) and deceptive (toward incorrect answers) contexts. We also find that LLM persuaders significantly increased quiz takers' accuracy, leading to higher earnings, when steering quiz takers toward correct answers, and significantly decreased their accuracy, leading to lower earnings, when steering them toward incorrect answers. Overall, our findings suggest that AI's persuasion capabilities already exceed those of humans that have real-money bonuses tied to performance. Our findings of increasingly capable AI persuaders thus underscore the urgency of emerging alignment and governance frameworks.

Problem

• Compare LLM and human persuasion in real-time conversations
• Assess LLM impact on quiz accuracy and earnings
• Highlight urgency for AI alignment and governance

Innovation

• LLM vs human persuaders in real-time quiz
• LLM achieves higher compliance in persuasion
• LLM impacts quiz accuracy and earnings significantly