Amnesia: Adversarial Semantic Layer Specific Activation Steering in Large Language Models

2026-03-10

AI Summary

Current safety alignment mechanisms for large language models—such as reinforcement learning from human feedback—are susceptible to circumvention and often fail to effectively suppress harmful content generation. This work proposes a lightweight activation-space adversarial attack that precisely manipulates semantic-specific activation states in intermediate Transformer layers, thereby inducing the model to generate malicious outputs like phishing emails or harmful code without altering model parameters or performing fine-tuning. The method achieves the first layer-specific activation intervention on open-source large language models and, when integrated into a red-teaming framework, substantially reduces the interception rate of existing safety mechanisms across multiple mainstream models. These findings expose critical vulnerabilities in current alignment strategies and underscore the urgent need to strengthen intrinsic model security.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Large language models for search
• Security and Privacy: Large-scale security measurements

Abstract

Warning: This article includes red-teaming experiments, which contain examples of compromised LLM responses that may be offensive or upsetting. Large Language Models (LLMs) have the potential to create harmful content, such as generating sophisticated phishing emails and assisting in writing code of harmful computer viruses. Thus, it is crucial to ensure their safe and responsible response generation. To reduce the risk of generating harmful or irresponsible content, researchers have developed techniques such as reinforcement learning with human feedback to align LLM's outputs with human values and preferences. However, it is still undetermined whether such measures are sufficient to prevent LLMs from generating interesting responses. In this study, we propose Amnesia, a lightweight activation-space adversarial attack that manipulates internal transformer states to bypass existing safety mechanisms in open-weight LLMs. Through experimental analysis on state-of-the-art, open-weight LLMs, we demonstrate that our attack effectively circumvents existing safeguards, enabling the generation of harmful content without the need for any fine-tuning or additional training. Our experiments on benchmark datasets show that the proposed attack can induce various antisocial behaviors in LLMs. These findings highlight the urgent need for more robust security measures in open-weight LLMs and underscore the importance of continued research to prevent their potential misuse.

Problem

• Large Language Models
• safety mechanisms
• harmful content
• adversarial attacks
• open-weight LLMs

Innovation

• adversarial attack
• activation steering
• large language models
• safety bypass
• semantic layer manipulation