Finding Safety Neurons in Large Language Models

2024-06-20

arXiv.org

AI Summary

Despite safety alignment, large language models (LLMs) still generate harmful content, and the underlying neural mechanisms remain poorly understood. Method: Leveraging mechanistic interpretability, this work systematically identifies sparse, efficient, and transferable “safety neurons” across models and datasets; it introduces activation contrast during generation and dynamic activation patching for causal validation. Contribution/Results: We find that safety and helpfulness partially share neuronal substrates but require distinct activation patterns—providing a neural explanation for the “alignment tax.” Experimentally, ablating only ~5% of identified safety neurons restores 90% of safety performance. The discovered neurons generalize robustly across diverse LLMs and red-teaming datasets. Moreover, they enable reliable pre-generation detection of unsafe outputs, offering a scalable, interpretable pathway toward safer LLM deployment.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) excel in various capabilities but also pose safety risks such as generating harmful content and misinformation, even after safety alignment. In this paper, we explore the inner mechanisms of safety alignment from the perspective of mechanistic interpretability, focusing on identifying and analyzing safety neurons within LLMs that are responsible for safety behaviors. We propose generation-time activation contrasting to locate these neurons and dynamic activation patching to evaluate their causal effects. Experiments on multiple recent LLMs show that: (1) Safety neurons are sparse and effective. We can restore $90$% safety performance with intervention only on about $5$% of all the neurons. (2) Safety neurons encode transferrable mechanisms. They exhibit consistent effectiveness on different red-teaming datasets. The finding of safety neurons also interprets"alignment tax". We observe that the identified key neurons for safety and helpfulness significantly overlap, but they require different activation patterns of the shared neurons. Furthermore, we demonstrate an application of safety neurons in detecting unsafe outputs before generation. Our findings may promote further research on understanding LLM alignment. The source codes will be publicly released to facilitate future research.

Problem

• Identifying safety neurons responsible for LLM safety behaviors
• Explaining alignment tax through overlapping safety-helpfulness neurons
• Developing methods to restore safety without affecting general capabilities

Innovation

• Identified safety neurons via activation contrasting
• Evaluated causal effects using dynamic activation patching
• Restored safety performance by patching neuron activations