Assessing the Potential of Generative Agents in Crowdsourced Fact-Checking

2025-04-24

AI Summary

This work investigates whether large language model (LLM)-driven generative agents can effectively replace human crowdworkers in fact-checking tasks. Method: We propose the first systematic framework integrating generative agents into crowdsourced fact-checking, simulating heterogeneous agent populations with diverse demographic and ideological profiles that autonomously perform evidence retrieval, multi-dimensional quality assessment (accuracy, precision, informativeness), and truth judgment. Contribution/Results: Experiments demonstrate that agents significantly outperform human crowdworkers in truth classification accuracy, exhibit higher internal consistency, substantially reduce sociocognitive biases, and base decisions more robustly on interpretable quality metrics. This study provides the first empirical validation of generative agents’ superiority and robustness in structured fact-checking, establishing a novel paradigm for automated misinformation governance.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: Crowd Sourcing and Human Computation
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The growing spread of online misinformation has created an urgent need for scalable, reliable fact-checking solutions. Crowdsourced fact-checking - where non-experts evaluate claim veracity - offers a cost-effective alternative to expert verification, despite concerns about variability in quality and bias. Encouraged by promising results in certain contexts, major platforms such as X (formerly Twitter), Facebook, and Instagram have begun shifting from centralized moderation to decentralized, crowd-based approaches. In parallel, advances in Large Language Models (LLMs) have shown strong performance across core fact-checking tasks, including claim detection and evidence evaluation. However, their potential role in crowdsourced workflows remains unexplored. This paper investigates whether LLM-powered generative agents - autonomous entities that emulate human behavior and decision-making - can meaningfully contribute to fact-checking tasks traditionally reserved for human crowds. Using the protocol of La Barbera et al. (2024), we simulate crowds of generative agents with diverse demographic and ideological profiles. Agents retrieve evidence, assess claims along multiple quality dimensions, and issue final veracity judgments. Our results show that agent crowds outperform human crowds in truthfulness classification, exhibit higher internal consistency, and show reduced susceptibility to social and cognitive biases. Compared to humans, agents rely more systematically on informative criteria such as Accuracy, Precision, and Informativeness, suggesting a more structured decision-making process. Overall, our findings highlight the potential of generative agents as scalable, consistent, and less biased contributors to crowd-based fact-checking systems.

Problem

• Assessing LLM-powered agents in crowdsourced fact-checking workflows
• Comparing generative agents' performance to human crowds in veracity judgments
• Evaluating agents' bias reduction and decision-making consistency in fact-checking

Innovation

• LLM-powered generative agents simulate human fact-checking
• Agents outperform humans in truthfulness and consistency
• Agents reduce bias using structured decision-making criteria