Large Language Models Generate Harmful Content Using a Distinct, Unified Mechanism

2026-04-10

AI Summary

Despite alignment training, large language models remain vulnerable to jailbreak attacks and can exhibit sudden misalignment after fine-tuning, yet the internal structural basis of their harmful mechanisms remains unclear. This work employs targeted weight pruning as a form of causal intervention to systematically probe the model’s internal machinery for generating harmful content. The study identifies a compact set of weights that are generalizable across diverse harm types and disentangled from benign capabilities. It reveals that alignment training reshapes internal representations by compressing these specific weights and elucidates the origin of abrupt misalignment. Furthermore, the research demonstrates that pruning these harmful weights substantially mitigates misaligned behavior, and crucially, shows that a model’s capacity to generate harmful content is independent of its ability to recognize such content.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Large language models (LLMs) undergo alignment training to avoid harmful behaviors, yet the resulting safeguards remain brittle: jailbreaks routinely bypass them, and fine-tuning on narrow domains can induce ``emergent misalignment''that generalizes broadly. Whether this brittleness reflects a fundamental lack of coherent internal organization for harmfulness remains unclear. Here we use targeted weight pruning as a causal intervention to probe the internal organization of harmfulness in LLMs. We find that harmful content generation depends on a compact set of weights that are general across harm types and distinct from benign capabilities. Aligned models exhibit a greater compression of harm generation weights than unaligned counterparts, indicating that alignment reshapes harmful representations internally--despite the brittleness of safety guardrails at the surface level. This compression explains emergent misalignment: if weights of harmful capabilities are compressed, fine-tuning that engages these weights in one domain can trigger broad misalignment. Consistent with this, pruning harm generation weights in a narrow domain substantially reduces emergent misalignment. Notably, LLMs harmful generation capability is dissociated from how they recognize and explain such content. Together, these results reveal a coherent internal structure for harmfulness in LLMs that may serve as a foundation for more principled approaches to safety.

Problem

• harmful content
• alignment
• emergent misalignment
• large language models
• safety

Innovation

• targeted weight pruning
• harm generation
• emergent misalignment
• internal representation
• alignment compression