Can Community Notes Replace Professional Fact-Checkers?

2025-02-19

AI Summary

This study investigates whether community notes on social media can substitute for professional fact-checking. Method: Leveraging large language models, we conducted large-scale semantic annotation, narrative linkage analysis, and source classification of community notes on Twitter/X. Contribution/Results: We quantitatively demonstrate—first empirically—that for posts propagating systemic misinformation narratives, community notes cite professional fact-checking sources five times more frequently than previously reported, and twice as often as other sources. These findings reveal a strong structural dependency: community notes rely heavily on professional fact-checkers for factual grounding and narrative framing. Removing professional oversight significantly degrades their efficacy. This work provides the first empirical evidence of such a hierarchical, interdependent relationship between community annotation and professional verification—directly challenging the prevailing “deprofessionalized substitution” hypothesis—and offers critical insights for designing collaborative, hybrid fact-checking mechanisms.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Application Domains: Misinformation & Fake News
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Social Networks and Social Media: Analysis of web-mediated communications, communities, and crowds
• Web Mining and Content Analysis: Community question answering

Abstract

Two commonly-employed strategies to combat the rise of misinformation on social media are (i) fact-checking by professional organisations and (ii) community moderation by platform users. Policy changes by Twitter/X and, more recently, Meta, signal a shift away from partnerships with fact-checking organisations and towards an increased reliance on crowdsourced community notes. However, the extent and nature of dependencies between fact-checking and helpful community notes remain unclear. To address these questions, we use language models to annotate a large corpus of Twitter/X community notes with attributes such as topic, cited sources, and whether they refute claims tied to broader misinformation narratives. Our analysis reveals that community notes cite fact-checking sources up to five times more than previously reported. Fact-checking is especially crucial for notes on posts linked to broader narratives, which are twice as likely to reference fact-checking sources compared to other sources. In conclusion, our results show that successful community moderation heavily relies on professional fact-checking.

Problem

• Evaluating community notes vs. professional fact-checking
• Analyzing dependency between community notes and fact-checking
• Assessing role of fact-checking in social media moderation

Innovation

• Language models annotate community notes
• Analyze dependencies on fact-checking sources
• Community notes rely on professional fact-checking