The Levers of Political Persuasion with Conversational AI

2025-07-18

AI Summary

This study investigates the trade-off between persuasive efficacy and factual accuracy of large language models (LLMs) across 707 political topics. Method: We conducted a three-stage, large-scale online experiment involving 76,977 participants, systematically evaluating 19 LLMs under diverse intervention strategies—including post-training, prompt engineering, personalization, and model scaling—and integrating automated fact-checking with multi-model cross-verification. Contribution/Results: Post-training and prompt engineering emerged as the most effective levers for enhancing persuasion (yielding up to a 51% improvement), substantially outperforming personalization and scale-alone approaches. However, this gain incurred a systematic decline in factual accuracy. To our knowledge, this is the first work to empirically quantify the persuasion–truthfulness trade-off in political discourse, establishing a methodological framework and providing critical empirical evidence for AI governance, human-AI interaction design, and the societal impact of LLMs on public deliberation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: Voting

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

There are widespread fears that conversational AI could soon exert unprecedented influence over human beliefs. Here, in three large-scale experiments (N=76,977), we deployed 19 LLMs-including some post-trained explicitly for persuasion-to evaluate their persuasiveness on 707 political issues. We then checked the factual accuracy of 466,769 resulting LLM claims. Contrary to popular concerns, we show that the persuasive power of current and near-future AI is likely to stem more from post-training and prompting methods-which boosted persuasiveness by as much as 51% and 27% respectively-than from personalization or increasing model scale. We further show that these methods increased persuasion by exploiting LLMs' unique ability to rapidly access and strategically deploy information and that, strikingly, where they increased AI persuasiveness they also systematically decreased factual accuracy.

Problem

• Evaluating persuasiveness of LLMs on political issues
• Assessing impact of post-training and prompting methods
• Analyzing trade-off between AI persuasiveness and factual accuracy

Innovation

• Post-training boosts persuasiveness by 51%
• Prompting methods increase persuasiveness by 27%
• Strategic information deployment reduces factual accuracy