Web(er) of Hate: A Survey on How Hate Speech Is Typed

2025-06-19

AI Summary

Hate speech dataset construction faces methodological trade-offs, with prevailing practices often compromising reliability for operational convenience. Method: We conduct a cross-dataset qualitative meta-analysis and methodological critique, systematically identifying twelve recurrent methodological pitfalls. We develop a reproducible evaluation matrix spanning annotation transparency, value positioning, and contextual modeling. Drawing on Max Weber’s “ideal type” theory, we propose a novel three-dimensional framework—value awareness, transparent annotation, and meta-methodological reflection—that integrates classical sociological theory into computational social science data methodology for the first time. Contribution/Results: This work shifts hate speech research from empirically driven practice toward reflexive scholarship, enhancing dataset rigor, interpretability, and ethical accountability. The framework provides actionable guidance for constructing socially responsible, theoretically grounded, and methodologically transparent hate speech datasets.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: Understanding People, Theories, Concepts and Methods

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The curation of hate speech datasets involves complex design decisions that balance competing priorities. This paper critically examines these methodological choices in a diverse range of datasets, highlighting common themes and practices, and their implications for dataset reliability. Drawing on Max Weber's notion of ideal types, we argue for a reflexive approach in dataset creation, urging researchers to acknowledge their own value judgments during dataset construction, fostering transparency and methodological rigour.

Problem

• Examines methodological choices in hate speech datasets
• Highlights implications for dataset reliability and practices
• Advocates reflexive approach and transparency in dataset creation

Innovation

• Critically examines hate speech dataset methodologies
• Advocates reflexive approach using Max Weber's ideal types
• Promotes transparency in dataset construction decisions