Towards Understanding the Robustness of Sparse Autoencoders

2026-04-20

AI Summary

This work addresses the vulnerability of large language models to optimization-based jailbreaking attacks and investigates the role of sparse autoencoders (SAEs) in enhancing robustness. The authors propose embedding a pretrained SAE into the Transformer residual stream during inference—without modifying model weights or blocking gradients—to improve defense capabilities. Experimental results reveal a monotonic negative correlation between L0 sparsity and attack success rate, and demonstrate that intermediate SAE layers effectively balance robustness and task performance, supporting the representational bottleneck hypothesis. Across multiple mainstream models, the SAE-augmented approach reduces jailbreak success rates by up to fivefold and significantly diminishes the transferability of cross-model attacks.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) remain vulnerable to optimization-based jailbreak attacks that exploit internal gradient structure. While Sparse Autoencoders (SAEs) are widely used for interpretability, their robustness implications remain underexplored. We present a study of integrating pretrained SAEs into transformer residual streams at inference time, without modifying model weights or blocking gradients. Across four model families (Gemma, LLaMA, Mistral, Qwen) and two strong white-box attacks (GCG, BEAST) plus three black-box benchmarks, SAE-augmented models achieve up to a 5x reduction in jailbreak success rate relative to the undefended baseline and reduce cross-model attack transferability. Parametric ablations reveal (i) a monotonic dose-response relationship between L0 sparsity and attack success rate, and (ii) a layer-dependent defense-utility tradeoff, where intermediate layers balance robustness and clean performance. These findings are consistent with a representational bottleneck hypothesis: sparse projection reshapes the optimization geometry exploited by jailbreak attacks.

Problem

• Large Language Models
• jailbreak attacks
• Sparse Autoencoders
• robustness
• gradient-based attacks

Innovation

• Sparse Autoencoders
• Jailbreak Robustness
• Representational Bottleneck
• L0 Sparsity
• Gradient-based Attacks