Lost in Moderation: How Commercial Content Moderation APIs Over- and Under-Moderate Group-Targeted Hate Speech and Linguistic Variations

2025-03-03

AI Summary

Commercial content moderation APIs exhibit systematic biases in detecting group-targeted hate speech—over-blocking anti-discrimination discourse and reclaimed slurs while under-detecting implicit hate, irony, and contextually benign uses of stigmatized terms. Method: We introduce the first reproducible auditing framework for black-box NLP systems, conducting 5 million API calls across four adversarial datasets. Our methodology integrates term-sensitive analysis with expert human annotation to rigorously evaluate five widely deployed commercial APIs. Contribution/Results: This study provides the first empirical characterization of misclassification mechanisms across LGBTQIA+, Black, Jewish, and Muslim linguistic contexts. We find all APIs rely heavily on identity-labeled tokens (e.g., “black”) as heuristic triggers, exhibiting severe deficits in contextual understanding and pragmatic reasoning. These findings establish a methodological benchmark and empirical foundation for assessing fairness and robustness in automated content moderation systems.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Commercial content moderation APIs are marketed as scalable solutions to combat online hate speech. However, the reliance on these APIs risks both silencing legitimate speech, called over-moderation, and failing to protect online platforms from harmful speech, known as under-moderation. To assess such risks, this paper introduces a framework for auditing black-box NLP systems. Using the framework, we systematically evaluate five widely used commercial content moderation APIs. Analyzing five million queries based on four datasets, we find that APIs frequently rely on group identity terms, such as ``black'', to predict hate speech. While OpenAI's and Amazon's services perform slightly better, all providers under-moderate implicit hate speech, which uses codified messages, especially against LGBTQIA+ individuals. Simultaneously, they over-moderate counter-speech, reclaimed slurs and content related to Black, LGBTQIA+, Jewish, and Muslim people. We recommend that API providers offer better guidance on API implementation and threshold setting and more transparency on their APIs' limitations. Warning: This paper contains offensive and hateful terms and concepts. We have chosen to reproduce these terms for reasons of transparency.

Problem

• Evaluates over- and under-moderation in commercial content moderation APIs.
• Assesses risks of silencing legitimate speech and missing harmful content.
• Proposes a framework to audit NLP systems for hate speech detection.

Innovation

• Framework for auditing black-box NLP systems
• Systematic evaluation of five commercial APIs
• Analysis of five million queries from datasets