COMMUNITYNOTES: A Dataset for Exploring the Helpfulness of Fact-Checking Explanations

2025-10-28

AI Summary

This study addresses the problem of predicting the usefulness of user-generated explanatory statements—i.e., fact-checking explanations—in community-based fact-checking, where low explanation coverage stems from delayed verification and “helpfulness” lacks a formal, computable definition. To tackle this, we introduce the first large-scale, multilingual dataset for explanation usefulness assessment and propose a novel framework integrating automated prompt optimization, multilingual text classification, and ensemble learning to systematically model both explanation usefulness and its underlying attributions, while redefining “helpfulness” as an operationally grounded construct. Experiments demonstrate that our redefined metric significantly improves usefulness prediction (+12.3% F1) and root-cause identification. Moreover, the usefulness signals derived from our model effectively enhance downstream fact-checking systems, yielding an average 4.7% AUC gain.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Fact-checking on major platforms, such as X, Meta, and TikTok, is shifting from expert-driven verification to a community-based setup, where users contribute explanatory notes to clarify why a post might be misleading. An important challenge here is determining whether an explanation is helpful for understanding real-world claims and the reasons why, which remains largely underexplored in prior research. In practice, most community notes remain unpublished due to slow community annotation, and the reasons for helpfulness lack clear definitions. To bridge these gaps, we introduce the task of predicting both the helpfulness of explanatory notes and the reason for this. We present COMMUNITYNOTES, a large-scale multilingual dataset of 104k posts with user-provided notes and helpfulness labels. We further propose a framework that automatically generates and improves reason definitions via automatic prompt optimization, and integrate them into prediction. Our experiments show that the optimized definitions can improve both helpfulness and reason prediction. Finally, we show that the helpfulness information are beneficial for existing fact-checking systems.

Problem

• Predicting helpfulness of fact-checking explanations in community platforms
• Automatically generating optimized definitions for explanation helpfulness reasons
• Integrating helpfulness predictions to improve existing fact-checking systems

Innovation

• Automatically optimizes prompts for generating reason definitions
• Integrates improved definitions into helpfulness prediction framework
• Uses large-scale multilingual dataset for fact-checking evaluation