Perceptions of Moderators as a Large-Scale Measure of Online Community Governance

2024-01-29

arXiv.org

AI Summary

Online community moderation lacks universally accepted, multidimensional evaluation criteria for assessing effectiveness. Method: Leveraging 1.89 million publicly posted user comments about moderators on Reddit, this study develops a fine-grained sentiment classification model (BERT fine-tuned) to quantify governance outcomes through users’ subjective perceptions. The approach integrates large-scale human annotation, causally inspired association analysis, and an anonymized open-source framework. Contributions/Results: (1) Governance strategy efficacy is topic-dependent—e.g., strict enforcement yields greater benefits in news-oriented subreddits; (2) “community-endogenous” moderators (i.e., long-standing, organically embedded members) significantly enhance user trust; (3) moderators’ pre-appointment participation strongly correlates with perceived credibility; (4) distinct subreddit categories exhibit significantly positive or negative sentiment toward moderation. The study releases the trained model, source code, and anonymized dataset to advance interpretable, reproducible research on community governance.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Application Domains: Humanities & Computational Social Science
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community

Application Category

• Web Mining and Content Analysis: Community question answering
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Millions of online communities are governed by volunteer moderators, who shape their communities by setting and enforcing rules, recruiting additional moderators, and participating in the community themselves. These moderators must regularly make decisions about how to govern, yet measuring the 'success' of governance is complex and nuanced, making it challenging to determine what governance strategies are most successful. Furthermore, prior work has shown that communities have differing values, suggesting that 'one-size-fits-all' approaches to governance are unlikely to serve all communities well. In this work, we assess governance practices on reddit by classifying the sentiment of community members' public discussion of their own moderators. We label 1.89 million posts and comments made on reddit over an 18 month period. We relate these perceptions to characteristics of community governance, and to different actions that community moderators take. We identify types of communities where moderators are perceived particularly positively and negatively, and highlight promising strategies for moderator teams. Amongst other findings, we show that strict rule enforcement is linked to more favorable perceptions of moderators of communities dedicated to certain topics, such as news communities, than others. We investigate what kinds of moderators are associated with improved community perceptions upon their addition to a mod team, and find that moderators who are active community members before and during their mod tenures are seen more favorably. We make our models, anonymized datasets, and code public.

Problem

• Online Community Moderation
• Evaluation Complexity
• Lack of General Standards

Innovation

• Community Management Styles
• Moderator Activity-Perception Correlation
• Model-and-Data Transparency