Effectiveness of Counter-Speech against Abusive Content: A Multidimensional Annotation and Classification Study

2025-06-13

AI Summary

This study addresses the challenge of evaluating the effectiveness of counter-speech in mitigating online hate speech. We propose the first sociologically grounded, six-dimensional evaluation framework—encompassing clarity, evidentiality, emotional appeal, and other theoretically motivated dimensions—and conduct collaborative human annotation of 4,214 counter-speech instances, resulting in the first large-scale, structured counter-speech dataset, publicly released. Methodologically, we introduce a novel multi-task learning architecture with dependency-aware classification to explicitly model inter-dimensional relationships. Our model achieves average F1 scores of 0.94 and 0.96 on development and test sets, respectively—significantly outperforming strong baselines—and demonstrates robust generalization across both expert-crafted and user-generated counter-speech. This work advances counter-speech evaluation from binary classification toward an interpretable, sociologically informed, and structurally rich paradigm.

Technology Category

• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Safety and Robustness
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Counter-speech (CS) is a key strategy for mitigating online Hate Speech (HS), yet defining the criteria to assess its effectiveness remains an open challenge. We propose a novel computational framework for CS effectiveness classification, grounded in social science concepts. Our framework defines six core dimensions - Clarity, Evidence, Emotional Appeal, Rebuttal, Audience Adaptation, and Fairness - which we use to annotate 4,214 CS instances from two benchmark datasets, resulting in a novel linguistic resource released to the community. In addition, we propose two classification strategies, multi-task and dependency-based, achieving strong results (0.94 and 0.96 average F1 respectively on both expert- and user-written CS), outperforming standard baselines, and revealing strong interdependence among dimensions.

Problem

• Defining criteria to assess counter-speech effectiveness against hate speech
• Proposing a computational framework for classifying counter-speech effectiveness
• Analyzing interdependence of six dimensions in counter-speech effectiveness

Innovation

• Multidimensional annotation framework for CS effectiveness
• Novel computational classification strategies
• Strong interdependence among effectiveness dimensions