Can (A)I Change Your Mind?

2025-03-03

AI Summary

This study investigates the capacity of large language models (LLMs) to influence human opinions in authentic, ecologically valid settings—specifically, cross-modal persuasion in non-English, unconstrained environments. Method: Conducting the first empirical study in Hebrew, we compare LLMs and human agents across two naturalistic interaction modalities—Telegram conversations and static text—on contentious public policy topics. A controlled experiment with 200 real users measures opinion adoption rates and confidence shifts via validated questionnaires. Contribution/Results: We find that LLMs exert persuasive influence comparable to humans; notably, beyond the static-text condition, LLMs significantly enhance users’ post-persuasion confidence. All interventions induce statistically significant attitude change. This work constitutes the first demonstration of LLM cross-modal persuasiveness outside English and laboratory contexts, establishing a novel paradigm and critical benchmark for assessing LLMs’ societal impact.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Application Domains: Humanities & Computational Social Science

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The increasing integration of large language model (LLM) based conversational agents into everyday life raises critical cognitive and social questions about their potential to influence human opinions. Although previous studies have shown that LLM-based agents can generate persuasive content, these typically involve controlled, English-language settings. Addressing this, our preregistered study explored LLM's persuasive capabilities in more ecological, unconstrained scenarios, examining both static (written paragraphs) and dynamic (conversations via Telegram) interaction types. Conducted entirely in Hebrew with 200 participants, the study assessed the persuasive effects of both LLM and human interlocutors on controversial civil policy topics. Results indicated that participants adopted LLM and human perspectives similarly, with significant opinion changes evident across all conditions, regardless of interlocutor type or interaction mode. Confidence levels increased significantly in most scenarios, except in static LLM interactions. These findings demonstrate LLM-based agents' robust persuasive capabilities across diverse sources and settings, highlighting their potential impact on shaping public opinions.

Problem

• Assessing LLM's persuasive impact in ecological, unconstrained scenarios beyond controlled settings
• Comparing LLM and human persuasion effectiveness on controversial civil policies
• Examining opinion change dynamics across static/digital interaction modes and confidence levels

Innovation

• Comparing static and dynamic LLM interactions in persuasion
• Testing LLM persuasion via Telegram in ecological scenarios
• Conducting Hebrew-language study with 200 participants scale