CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification

2026-04-16

AI Summary

This work addresses the challenge that large language models often generate toxic content, and existing detoxification methods frequently compromise generation quality or rely on costly human annotations. Leveraging causal inference, the authors propose using the Probability of Necessity and Sufficiency (PNS) to identify attention heads critically responsible for toxicity generation. They introduce two efficient detoxification strategies: input-aware dynamic intervention during inference and PNS-guided fine-tuning, which permanently eliminate toxic representations. Additionally, they construct a new benchmark, PARATOX, to support counterfactual evaluation. Experiments demonstrate that their approach outperforms baseline methods by 5.34% in toxicity reduction on datasets such as ToxiGen while preserving linguistic fluency, and accelerates causal head selection by a factor of seven.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Causal Learning
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) frequently generate toxic content, posing significant risks for safe deployment. Current mitigation strategies often degrade generation quality or require costly human annotation. We propose CAUSALDETOX, a framework that identifies and intervenes on the specific attention heads causally responsible for toxic generation. Using the Probability of Necessity and Sufficiency (PNS), we isolate a minimal set of heads that are necessary and sufficient for toxicity. We utilize these components via two complementary strategies: (1) Local Inference-Time Intervention, which constructs dynamic, input-specific steering vectors for context-aware detoxification, and (2) PNS-Guided Fine-Tuning, which permanently unlearns toxic representations. We also introduce PARATOX, a novel benchmark of aligned toxic/non-toxic sentence pairs enabling controlled counterfactual evaluation. Experiments on ToxiGen, ImplicitHate, and ParaDetox show that CAUSALDETOX achieves up to 5.34% greater toxicity reduction compared to baselines while preserving linguistic fluency, and offers a 7x speedup in head selection.

Problem

• toxic content
• language model detoxification
• safe deployment
• toxic generation

Innovation

• Causal Intervention
• Attention Head Selection
• Language Model Detoxification
• PNS
• Counterfactual Evaluation