Dealing with Annotator Disagreement in Hate Speech Classification

2025-02-12

AI Summary

This work addresses severe and long-overlooked inter-annotator subjectivity disagreement in Turkish tweet hate speech classification. We propose a BERT fine-tuning framework integrating multi-annotator consistency modeling, soft-label learning, and uncertainty-aware training. To our knowledge, this is the first systematic evaluation of disagreement-aware modeling for non-English hate speech detection, and we introduce the first standardized multi-annotator benchmark for this task. Our method explicitly models annotation uncertainty, treating annotator disagreement as structured supervisory signal rather than noise—thereby enhancing model robustness and generalization. Evaluated on a real-world Turkish tweet dataset with multiple annotations per instance, our approach achieves state-of-the-art performance, significantly outperforming single-annotator baselines in accuracy. The framework offers a transferable methodological paradigm for low-resource, high-subjectivity text classification tasks.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Uncertainty Representations

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Hate speech detection is a crucial task, especially on social media, where harmful content can spread quickly. Implementing machine learning models to automatically identify and address hate speech is essential for mitigating its impact and preventing its proliferation. The first step in developing an effective hate speech detection model is to acquire a high-quality dataset for training. Labeled data is foundational for most natural language processing tasks, but categorizing hate speech is difficult due to the diverse and often subjective nature of hate speech, which can lead to varying interpretations and disagreements among annotators. This paper examines strategies for addressing annotator disagreement, an issue that has been largely overlooked. In particular, we evaluate different approaches to deal with annotator disagreement regarding hate speech classification in Turkish tweets, based on a fine-tuned BERT model. Our work highlights the importance of the problem and provides state-of-art benchmark results for detection and understanding of hate speech in online discourse.

Problem

• Addressing annotator disagreement in hate speech classification.
• Improving hate speech detection in Turkish tweets using BERT.
• Developing strategies for high-quality hate speech datasets.

Innovation

• BERT model fine-tuning
• Annotator disagreement strategies
• Turkish tweets classification