A Hybrid Theory and Data-driven Approach to Persuasion Detection with Large Language Models

2025-11-26

AI Summary

To address the limitation of traditional belief revision models in capturing large-scale textual persuasion processes within social media environments, this study integrates psychological theory with large language models (LLMs) to develop an interpretable online persuasion detection framework. Methodologically, we design psychology-informed experiments to extract two key predictive factors—“cognitive emotion” and “sharing intention”—and leverage LLMs to generate fine-grained psychological feature scores, which are then fed into a random forest classifier to predict individual belief change. Compared to purely data-driven approaches, this hybrid paradigm significantly improves prediction accuracy, empirically confirming cognitive emotion and sharing intention as the strongest persuasion indicators. Our primary contribution is the first systematic incorporation of cognitive emotion mechanisms into LLM-based persuasion modeling, thereby reconciling theoretical interpretability with data-driven performance. This work provides a novel analytical tool and empirical foundation for influence assessment, misinformation detection, and narrative efficacy evaluation.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Traditional psychological models of belief revision focus on face-to-face interactions, but with the rise of social media, more effective models are needed to capture belief revision at scale, in this rich text-based online discourse. Here, we use a hybrid approach, utilizing large language models (LLMs) to develop a model that predicts successful persuasion using features derived from psychological experiments. Our approach leverages LLM generated ratings of features previously examined in the literature to build a random forest classification model that predicts whether a message will result in belief change. Of the eight features tested, extit{epistemic emotion} and extit{willingness to share} were the top-ranking predictors of belief change in the model. Our findings provide insights into the characteristics of persuasive messages and demonstrate how LLMs can enhance models of successful persuasion based on psychological theory. Given these insights, this work has broader applications in fields such as online influence detection and misinformation mitigation, as well as measuring the effectiveness of online narratives.

Problem

• Develops a model predicting belief change from persuasive messages
• Identifies key psychological features like epistemic emotion and sharing willingness
• Applies large language models to enhance persuasion detection in online discourse

Innovation

• Hybrid approach combining theory and data-driven methods
• LLM-generated ratings for psychological feature extraction
• Random forest classification predicting belief change likelihood