Evaluating the Capabilities of LLMs for Persuasive Dialogue

2026-08-30

AI Summary

研究通过构建多代理对话平台Persuasio，评估了大型语言模型在说服性对话中的表现，揭示了修辞流畅性和形式论证强度之间的系统性差距。

Abstract

Large language models (LLMs) can generate apparently highly persuasive text, but does sounding persuasive mean arguing well? We introduce \textsc{Persuasio}, a multi-agent dialogue platform grounded in a formal argumentation-based theory of persuasion dialogues that adjudicates logical winners during free-text debates. Using this system, we generated 192 debates on a UK political topic between humans and LLMs, and evaluated 22 interlocutors through both automated adjudication and 9,702 crowdsourced pairwise judgements across 1{,}386 annotation instances. We observed a consistent decoupling between subjective and formal persuasiveness: LLMs dominated the subjective ranking yet performed substantially worse under argumentation-theoretic adjudication, where humans remained competitive. Multi-agent and retrieval-augmented variants further widened this divergence. These findings reveal a systematic gap between rhetorical fluency and formal argumentative strength in LLM-based persuasive dialogues.

Problem

• LLMs
• Persuasive Dialogue
• Argumentation

Innovation

• multi-agent dialogue platform
• formal argumentation-based theory
• automated adjudication
• crowdsourced pairwise judgements