A Survey on Automatic Credibility Assessment of Textual Credibility Signals in the Era of Large Language Models

2024-10-28

arXiv.org

AI Summary

The proliferation of social media and generative AI has intensified the need for automated content credibility assessment; however, existing research remains fragmented, lacking systematic integration and unified modeling of multi-source credibility signals—including factual accuracy, bias, persuasive techniques, logical fallacies, and claim veracity. This paper systematically reviews 175 studies and introduces the first comprehensive taxonomy covering nine categories of textual credibility signals, organized along four core dimensions: factual accuracy, subjectivity/bias, persuasion/logical fallacies, and claim veracity. We propose a novel LLM-driven credibility assessment paradigm that synergistically integrates NLP, explainable AI, multidimensional signal modeling, and advanced prompting/fine-tuning techniques. Our work establishes the most comprehensive literature map of credibility signal assessment to date, identifies key technical bottlenecks, and reveals new opportunities enabled by generative AI—thereby providing both a theoretical framework and practical guidelines for trustworthy NLP.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

In the current era of social media and generative AI, an ability to automatically assess the credibility of online social media content is of tremendous importance. Credibility assessment is fundamentally based on aggregating credibility signals, which refer to small units of information, such as content factuality, bias, or a presence of persuasion techniques, into an overall credibility score. Credibility signals provide a more granular, more easily explainable and widely utilizable information in contrast to currently predominant fake news detection, which utilizes various (mostly latent) features. A growing body of research on automatic credibility assessment and detection of credibility signals can be characterized as highly fragmented and lacking mutual interconnections. This issue is even more prominent due to a lack of an up-to-date overview of research works on automatic credibility assessment. In this survey, we provide such systematic and comprehensive literature review of 175 research papers while focusing on textual credibility signals and Natural Language Processing (NLP), which undergoes a significant advancement due to Large Language Models (LLMs). While positioning the NLP research into the context of other multidisciplinary research works, we tackle with approaches for credibility assessment as well as with 9 categories of credibility signals (we provide a thorough analysis for 3 of them, namely: 1) factuality, subjectivity and bias, 2) persuasion techniques and logical fallacies, and 3) claims and veracity). Following the description of the existing methods, datasets and tools, we identify future challenges and opportunities, while paying a specific attention to recent rapid development of generative AI.

Problem

• Automatically assessing online content credibility using textual signals
• Integrating fragmented research on multiple credibility signal detection
• Providing comprehensive overview of NLP credibility assessment methods

Innovation

• Systematic literature review of 175 NLP papers
• Analyzes nine categories of textual credibility signals
• Focuses on factuality, persuasion techniques, and bias