SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behavior

2026-06-16

AI Summary

This work addresses the limitations of sparse autoencoder (SAE) interventions in suppressing harmful behaviors, which can be undermined by post-intervention behavioral recovery. We systematically uncover, for the first time, the inconsistency between SAE feature-level interventions and actual behavioral control, introducing the concept of “post-intervention recovery” and establishing a rigorous evaluation framework under a strict threat model. By integrating residual space constraint optimization, orthogonal encoder updates, Jacobian analysis of feature maps, and attribution of recovery pathways, our approach achieves a 95.8% behavioral recovery rate on safety-critical tasks such as refusal elicitation, while maintaining precise clamping of target features (with drift as low as 0.131), substantially outperforming baseline methods.

Abstract

Sparse Autoencoders (SAEs) decompose residual-stream activations into interpretable features. Recent latent-space defenses increasingly rely on these decompositions, assuming that identified "unsafe" SAE features serve as actionable handles for monitoring and intervention. In this paradigm, clamping a specific harmful feature is expected to reliably prevent model misbehavior. However, we show that this success may hide a recoverable failure mode: the clamp may block one visible route to a behavior without eliminating the behavior itself. We formulate this vulnerability as post-intervention recovery, a constrained residual-space optimization problem. Starting from the post-intervention residual state, we optimize residual perturbations to recover the pre-intervention behavior while preserving the post-intervention values of the targeted SAE features. Even under a strong threat model where the intervention remains active throughout optimization and generation, recovery remains possible. To rule out that recovery simply undoes the intervention, we use encoder-orthogonal updates for single-layer interventions and the corresponding feature-map Jacobian in the cross-layer setting. Across TPP, unlearning, IOI, and refusal steering experiments, this stress test reveals recoverable behavior despite successful feature-level intervention. Especially in the safety-critical refusal-steering setting, we achieve a 95.8% recovery rate on valid samples while keeping defended-feature relative drift to 0.131, substantially below suffix-based baselines. A recovery-path attribution analysis further localizes this recovery to the SAE reconstruction residual, the component left unexplained by the SAE. These results expose a gap between feature-level control and behavioral completeness: SAE features can support causal intervention, but controlling them does not guarantee control over the underlying behavior.

Problem

• Sparse Autoencoders
• post-intervention recovery
• feature-level intervention
• behavioral completeness
• residual-stream activations

Innovation

• Sparse Autoencoders
• Post-intervention Recovery
• Residual-space Optimization
• Feature-level Intervention
• Behavioral Completeness