From Birdwatch to Community Notes, from Twitter to X: four years of community-based content moderation

2025-10-10

AI Summary

This study systematically evaluates the effectiveness and evolutionary dynamics of X’s Community Notes crowdsourced content moderation system. Addressing the lack of long-term, structured empirical data in prior work, we construct the first large-scale, open-source dataset covering 2020–2024, featuring multilingual detection, fine-grained topic classification, URL semantic extraction, and monthly user collaboration networks. Methodologically, we integrate NLP, language identification, dynamic topic modeling, and complex network analysis, complemented by a systematic literature review. Our key contributions are: (1) releasing the first four-year structured Community Notes dataset alongside a complete, reproducible analytical toolchain; (2) uncovering cross-lingual and cross-topical mechanisms of consensus formation and bias patterns in community moderation; and (3) establishing a replicable empirical framework and standardized research infrastructure for trustworthy platform content governance.

Technology Category

• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Application Domains: Humanities & Computational Social Science
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Web Mining and Content Analysis: Community question answering
• Social Networks and Social Media: Analysis of web-mediated communications, communities, and crowds
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Community Notes (formerly known as Birdwatch) is the first large-scale crowdsourced content moderation initiative that was launched by X (formerly known as Twitter) in January 2021. As the Community Notes model gains momentum across other social media platforms, there is a growing need to assess its underlying dynamics and effectiveness. This Resource paper provides (a) a systematic review of the literature on Community Notes, and (b) a major curated dataset and accompanying source code to support future research on Community Notes. We parsed Notes and Ratings data from the first four years of the program and conducted language detection across all Notes. Focusing on English-language Notes, we extracted embedded URLs and identified discussion topics in each Note. Additionally, we constructed monthly interaction networks among the Contributors. Together with the literature review, these resources offer a robust foundation for advancing research on the Community Notes system.

Problem

• Evaluating effectiveness of crowdsourced content moderation systems
• Analyzing dynamics of Community Notes program on social media
• Providing datasets for research on community-based fact checking

Innovation

• Large-scale crowdsourced content moderation system
• Parsed Notes and Ratings data over four years
• Constructed monthly interaction networks among Contributors