Concept-Based Interpretability for Toxicity Detection

2025-11-15

AI Summary

This work addresses the lack of concept-level interpretability and imbalanced concept attribution—leading to misclassifications—in toxic language detection. Methodologically: (1) it treats semantic subtypes (e.g., insult, threat, identity attack) as interpretable concepts, constructs a target lexicon, and proposes a Word–Concept Alignment (WCA) score to quantify each token’s contribution to misclassification via concept gradients (CG); (2) it introduces, for the first time, a delexicalized generative data augmentation strategy to assess model reliance on abstract toxic patterns rather than surface lexical cues. Experiments demonstrate that CG precisely identifies critical toxic tokens and reveal that models over-attribute toxicity to conceptual features even when explicit toxic words are absent—exposing systematic generalization biases toward deep semantic patterns and implicit dependency mechanisms. This establishes a novel paradigm for interpretable toxic language detection grounded in concept-level attribution and causal probing.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Transparent, Interpretable, Explainable ML
• Computer Vision: Language and Vision

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The rise of social networks has not only facilitated communication but also allowed the spread of harmful content. Although significant advances have been made in detecting toxic language in textual data, the exploration of concept-based explanations in toxicity detection remains limited. In this study, we leverage various subtype attributes present in toxicity detection datasets, such as obscene, threat, insult, identity attack, and sexual explicit as concepts that serve as strong indicators to identify whether language is toxic. However, disproportionate attribution of concepts towards the target class often results in classification errors. Our work introduces an interpretability technique based on the Concept Gradient (CG) method which provides a more causal interpretation by measuring how changes in concepts directly affect the output of the model. This is an extension of traditional gradient-based methods in machine learning, which often focus solely on input features. We propose the curation of Targeted Lexicon Set, which captures toxic words that contribute to misclassifications in text classification models. To assess the significance of these lexicon sets in misclassification, we compute Word-Concept Alignment (WCA) scores, which quantify the extent to which these words lead to errors due to over-attribution to toxic concepts. Finally, we introduce a lexicon-free augmentation strategy by generating toxic samples that exclude predefined toxic lexicon sets. This approach allows us to examine whether over-attribution persists when explicit lexical overlap is removed, providing insights into the model's attribution on broader toxic language patterns.

Problem

• Developing concept-based interpretability for toxicity detection models
• Addressing misclassification caused by disproportionate concept attribution
• Analyzing model behavior when explicit toxic lexicons are removed

Innovation

• Concept Gradient method for causal interpretation
• Targeted Lexicon Set to capture misclassification words
• Lexicon-free augmentation to test over-attribution