It is not enough to give your moderation rules to ChatGPT: Policy-as-Prompt Moderation and Its Potential Impacts on Community Governance

2026-07-13

AI Summary

This study investigates the governance risks arising from directly encoding content moderation policies as natural language prompts into large language models (LLMs), with a focus on the limitations of the “Policy-as-Prompt” approach in decentralized community moderation. Integrating LLM prompt engineering, community governance theory, and content moderation practice, this work systematically demonstrates that reliance on prompts alone is insufficient for effective governance. It further identifies adverse impacts on policy enforcement accuracy, community autonomy, and moderator well-being. To address these shortcomings, the research proposes a holistic governance framework that transcends prompt-based methods by incorporating transparency mechanisms, user feedback loops, and multi-stakeholder participation, thereby offering a novel pathway toward synergistic socio-technical systems for content moderation.

Abstract

Content moderation practices and governance paradigms are changing rapidly, as fewer human moderators are deployed as `experts' by social media companies in a centralized manner. Instead, the companies are focusing more on community approaches, relying on volunteers to provide accurate information and make correct decisions. In decentralized moderation, communities have always relied on volunteers, updated community guidelines, and internal discussions thereof. For both content moderation paradigms, Artificial Intelligence (AI) seems like it could help ease moderation burdens of time, mental health, and accuracy. One possible way to operationalize AI in content moderation is a `policy-as-prompt'' approach, where the policy is formulated as a natural-language prompt and then passed to a large language model (LLM). This model then aids in moderation tasks. In this paper, we briefly lay out the technical and governance properties of this approach, and argue that its limitations lead to specific risks and harms that have to be addressed. Towards alleviating them, we lay out multiple considerations towards more effective prompt governance, but ultimately find that writing prompts alone is not appropriate for ensuring meaningful community governance.

Problem

• content moderation
• policy-as-prompt
• community governance
• large language models
• AI ethics

Innovation

• Policy-as-Prompt
• content moderation
• large language models
• community governance
• prompt engineering