Statistically Grounded Sparse-Feature Interventions for Activation-Space Control in Large Language Models

2026-06-05

AI Summary

This work addresses the challenge of achieving high-quality, auditable behavioral control in large language models without fine-tuning. The authors propose a novel optimization-free activation steering method that leverages sparse autoencoders to extract semantic features and employs a transparent, interpretable feature selection and weighting mechanism. This mechanism integrates six reliability screening criteria with a Borda consensus ranking derived from three statistical measures: F-test, KSG mutual information, and Cohen’s d. Demonstrating the first fully auditable, non-optimization-based intervention, the approach matches the performance of fine-tuning methods across four behavioral domains on Gemma-series models while significantly outperforming existing baselines. Crucially, it effectively modulates model behavior without compromising output relevance, richness, or coherence.

Abstract

Activation steering offers a lightweight alternative to fine-tuning for behavioral control of large language models, but SAE-based steering methods often rely on learned steering objectives or single-criterion feature selection. We introduce a transparent SAE-feature steering pipeline that first applies a six-condition reliability filter, then ranks sparse features through an unweighted Borda consensus over three complementary statistics: $F$-test, KSG mutual information, and Cohen's $d$. The resulting steering direction is constructed as a Cohen's-$d$-weighted combination of SAE decoder rows, providing an optimization-free direction motivated by Fisher-LDA under approximate SAE-feature decorrelation. Across three Gemma-family models, four behavioral domains, and 356 layer-strength configurations, the method produces measurable domain-specific shifts while revealing a substantial gap between raw attribute movement and quality-preserving generation. In the strongest configuration, logical-correctness steering reaches a primary-score delta of $+1.16$ in Gemma~2 9B; however, our broader finding is that usable steering is highly localized by model, domain, layer, and strength. These results argue that activation-steering evaluations should report quality-conditioned success alongside raw behavioral shift. Our code and data are available at https://github.com/Oshayer-Siddique/LLM-Steering-Using-SAE.

Problem

• activation steering
• feature selection
• large language models
• statistical consensus
• sparse autoencoders

Innovation

• activation steering
• sparse autoencoders
• optimization-free
• statistical consensus
• feature selection