How Human Feedback Shapes AI-generated Community Notes

2026-06-29

AI Summary

This study investigates how artificial intelligence in the crowdsourced fact-checking system Community Notes refines its generated content through human feedback. Leveraging a dataset of 19,146 collaborative notes and 211,850 human feedback instances, the authors develop a typology of feedback categories and employ large-scale log analysis, content comparison, and user engagement correlation to systematically uncover the mechanisms by which human feedback improves AI-generated content. The findings reveal that feedback challenging the main claims of a note most significantly enhances its perceived usefulness. Although collaborative notes exhibit lower adoption rates, they effectively address misinformation overlooked by purely human- or AI-authored notes, thereby playing a crucial complementary role in the fact-checking ecosystem.

Abstract

Community Notes, a bridging-based crowd-sourced fact-checking system, has emerged as a new mechanism for moderating misleading information on social media and has been adopted by major platforms including X, Facebook, Instagram, Threads, and TikTok. Since its introduction, there has been an open question about what role AI could play in scaling and optimizing the system. Recently, X extended its Community Notes system by introducing Collaborative Notes: notes initially drafted by an LLM and iteratively refined based on feedback from human contributors. In this work, we systematically analyze the complete corpus of 19,146 collaborative notes and 211,850 instances of human feedback. First, we develop a taxonomy of human suggestions for improving AI-generated note drafts and find that suggestions involving factual corrections and additional context are most likely to be incorporated, while subjective policy judgments rarely are. Second, we examine changes in helpfulness across versions of collaborative notes and find that human feedback leads to more helpful notes, with the greatest impact coming from suggestions that challenge the main claim in the previous draft, particularly when submitted by more active contributors. Finally, we find that although collaborative notes improve through human feedback, they reach helpful status and are shown on the platform at lower rates than human-only or AI-only notes, with limited human participation emerging as a key bottleneck. Nevertheless, rather than serving as a weaker substitute, collaborative notes tend to play a complementary role, predominantly targeting posts that do not attract human-only or AI-only notes. Our analysis provides an initial description of efforts to use AI to improve crowdsourced content moderation in a real-world moderation system and outlines pathways for future improvements to such features.

Problem

• Human Feedback
• AI-generated Content
• Community Notes
• Content Moderation
• Collaborative Fact-checking

Innovation

• human-AI collaboration
• crowdsourced fact-checking
• large language models
• content moderation
• human feedback