Commenotes: Synthesizing Organic Comments to Support Community-Based Fact-Checking

2025-09-13

AI Summary

Community fact-checking often fails due to latency, despite early debunking signals already present in user comments. This paper addresses this problem by proposing Commenotes, a two-stage framework that systematically validates— for the first time—that 99.3% of misleading posts yield effective debunking comments within two hours of publication. Leveraging a corpus of 2.2 million social media replies, Commenotes integrates comment filtering with controllable natural language generation to synthesize high-quality, high-credibility composite annotations (“commenotes”). A user study demonstrates that generated commenotes achieve user trust in 85.8% of cases and outperform human-written annotations by 70.1% in perceived helpfulness. This work establishes a novel, community-driven, real-time fact-checking paradigm—“verification via commentary”—where early user engagement directly informs and accelerates authoritative correction.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Application Domains: Misinformation & Fake News
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Community question answering
• Social Networks and Social Media: Social mining and social search on the Web

Abstract

Community-based fact-checking is promising to reduce the spread of misleading posts at scale. However, its effectiveness can be undermined by the delays in fact-check delivery. Notably, user-initiated organic comments often contain debunking information and have the potential to help mitigate this limitation. Here, we investigate the feasibility of synthesizing comments to generate timely high-quality fact-checks. To this end, we analyze over 2.2 million replies on X and introduce Commenotes, a two-phase framework that filters and synthesizes comments to facilitate fact-check delivery. Our framework reveals that fact-checking comments appear early and sufficiently: 99.3% of misleading posts receive debunking comments within the initial two hours since post publication, with synthesized extit{commenotes} successfully earning user trust for 85.8% of those posts. Additionally, a user study (N=144) found that the synthesized commenotes were often preferred, with the best-performing model achieving a 70.1% win rate over human notes and being rated as significantly more helpful.

Problem

• Synthesizing organic comments for timely fact-checking
• Reducing delays in community-based fact-checking delivery
• Generating user-trusted debunking information from comments

Innovation

• Two-phase framework filtering and synthesizing comments
• Generating timely high-quality fact-checks from organic replies
• Synthesized commenotes achieving user trust and preference