A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models

2026-05-08

AI Summary

It remains unclear whether the safety alignment mechanisms of large language models are robust and whether they rely on distributed representations or critical individual neurons. This work addresses this gap by identifying and intervening on “refusal neurons” and “concept neurons,” demonstrating that the safety mechanisms of seven prominent models (ranging from 1.7B to 70B parameters) can be bypassed bidirectionally without any retraining or prompt engineering. The study reveals, for the first time, that safety alignment is governed by a small set of causally sufficient neurons rather than being diffusely distributed across the network—challenging prevailing assumptions about alignment robustness. Experiments show that suppressing just a single refusal neuron can cause models to comply with diverse harmful requests, and this vulnerability exhibits cross-model transferability.

Abstract

Safety alignment in language models operates through two mechanistically distinct systems: refusal neurons that gate whether harmful knowledge is expressed, and concept neurons that encode the harmful knowledge itself. By targeting a single neuron in each system, we demonstrate both directions of failure -- bypassing safety on explicit harmful requests via suppression, and inducing harmful content from innocent prompts via amplification -- across seven models spanning two families and 1.7B to 70B parameters, without any training or prompt engineering. Our findings suggest that safety alignment is not robustly distributed across model weights but is mediated by individual neurons that are each causally sufficient to gate refusal behavior -- suppressing any one of the identified refusal neurons bypasses safety alignment across diverse harmful requests.

Problem

• safety alignment
• refusal neurons
• concept neurons
• large language models
• adversarial robustness

Innovation

• refusal neurons
• concept neurons
• safety alignment
• neuron suppression
• large language models