LLMs can persuade only psychologically susceptible humans on societal issues, via trust in AI and emotional appeals, amid logical fallacies

2026-04-18

AI Summary

This study addresses the lack of longitudinal evidence and psychological mechanisms underlying how large language models (LLMs) influence human attitudes on polarized societal issues. Using the Talk2AI longitudinal experimental framework, 770 participants engaged in multi-round dialogues with four leading LLMs on contentious topics. Integrating psychological, reasoning, and affective dimensions, the research employed multiverse analyses, natural language processing, explainable AI, and mixed-effects modeling. It reveals that only psychologically susceptible individuals—those high in trust, agreeableness, extraversion, and need for cognition—are significantly influenced by LLMs. Persuasive effects stem from AI trust and emotional appeals rather than logical superiority, challenging the stereotype of LLMs’ rational dominance. The study also demonstrates attitudinal inertia in humans, frequent use of logical fallacies by both humans and LLMs, and shows that anthropomorphic perceptions of LLMs can be effectively predicted by psychological and behavioral features (R² up to 0.44).

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: (Large) Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Scarce longitudinal evidence examines LLMs' persuasiveness and humanness along time-evolving psychological frameworks. We introduce Talk2AI, a longitudinal framework quantifying psycho-social, reasoning and affective dimensions of LLMs' persuasiveness about polarizing societal topics. In a four-way longitudinal setup, Talk2AI's 770 participants engaged in structured conversations with one of four leading LLMs on topics like climate change, social media misinformation, and math anxiety. This produced 3,080 conversations over 60,000 turns. After each wave, participants reported conviction in their initial topic stance, perceived opinion change, LLM's perceived humanness, a self-donation to the topic and a textual explanation. Feedback time series showed longitudinal inertia in convictions, indicating some human anchoring to initial opinions even after repeated exposure to AI-generated arguments. Interestingly, NLP analyses revealed that both humans and LLMs relied on fallacious reasoning in 1 conversational quip every 6, countering the ``LLMs as superior systems" stereotype behind LLMs' cognitive surrender. LLMs' perceived humanness was most learnable from sociodemographic, psychological and engagement features ($R^2=0.44$), followed by opinion change ($R^2=0.34$), conviction ($R^2=0.26$) and personal endowment ($R^2=0.24$). Crucially, explainable AI (XAI) indicated: (i) the presence of individuals more susceptible to LLM-based opinion changes; (ii) psychological susceptibility to LLM-convincing consisted of having more trust in LLMs, being more agreeable and extraverted and with a higher need for cognition. A multiverse approach with mixed-effects models confirmed XAI results, alongside strong individual differences. Talk2AI provides a grounded framework and evidence for detecting how GenAI can influence human opinions via multiple psycho-social pathways in AI-human digital platforms.

Problem

• LLM persuasion
• psychological susceptibility
• longitudinal study
• opinion change
• human-AI interaction

Innovation

• longitudinal framework
• explainable AI (XAI)
• psychological susceptibility
• fallacious reasoning
• perceived humanness