Localizing Safety Alignment: MLP Layers and Mid-Network Blocks Encode Refusal Behavior in Large Language Models

2026-08-11

AI Summary

This study investigates whether refusal behaviors in large language models—arising from safety alignment—are governed by specific model parameters. Through fine-grained weight transplantation between aligned and base models—including attention modules, MLPs, contiguous layers, and MLP sub-blocks—combined with greedy decoding and multi-benchmark evaluation, the authors find that refusal behavior is predominantly encoded in MLP weights of middle network layers (e.g., layers 8–11), contributing over 2.7 times more than attention mechanisms. The work further reveals that safety-related components exhibit non-additive effects and benchmark dependency. Notably, selectively transplanting key MLP sub-blocks achieves high refusal rates while mitigating over-refusal risks, outperforming full MLP replacement.

Abstract

Safety alignment in large language models is often treated as a distributed property of the entire network, yet its practical brittleness suggests that refusal behavior may be concentrated in a smaller set of parameters. This work addresses where safety-aligned refusal is encoded by transplanting weights from aligned models into matched unaligned base models at multiple levels of granularity. Using two open-weight model pairs and four safety benchmarks, we conducted experiments to compare the effects of replacing attention weights, MLP weights, contiguous layer regions, and MLP blocks. Across both model families, refusal transfer is dominated by MLP weights: replacing MLP parameters recovers substantially more malicious-prompt refusal than replacing attention parameters, with gains of at least 2.7 times more across benchmarks. Within the MLP stack, refusal-relevant parameters exhibit a consistent mid-network concentration, as the block spanning layers 8-11 is selected first in all six greedy searches over model-dataset pairs. The results also show that the composition of safety-relevant components is non-additive: in five of six greedy trajectories, adding more aligned blocks can reduce refusal performance, and selective block subsets can outperform full MLP transplantation on malicious refusal, benign over-refusal, or both. Finally, greedy orders transferred to OR-Bench vary with the source benchmark used to derive them, indicating a benchmark-dependent precision-coverage trade-off. These results suggest that safety alignment in current LLMs is both localized and interaction-sensitive, offering insight into alignment brittleness and potential avenues for targeted safety interventions.

Problem

• safety alignment
• refusal behavior
• large language models
• MLP layers
• parameter localization

Innovation

• safety alignment
• MLP localization
• refusal behavior
• parameter transplantation
• non-additive composition