A Coin Flip Per Token: Bernoulli Sparse Steering of Large Language Models

2026-07-06

AI Summary

Existing activation intervention methods based on sparse autoencoders (SAEs) apply uniform perturbations to every token, often degrading linguistic fluency. This work proposes Stochastic Token Sparsification (STS) and Stochastic Block Sparsification (SBS), which leverage Bernoulli sampling to sparsely intervene on only a subset of tokens, achieving effective behavioral control without requiring reward models or learnable gating mechanisms. The study reveals, for the first time, that the efficacy of SAE-based interventions depends on the cumulative “signal dose” across a sequence, demonstrating that intervening on merely 30%–50% of tokens can match or even surpass the performance of dense intervention or prompt engineering. Experiments across two model families and tasks show that a 50% intervention rate recovers most of the performance of dense intervention while preserving fluency, and even a 30% rate significantly outperforms prompt-based control.

Abstract

Activation steering via sparse autoencoders (SAEs) enables behavioral control of large language models without task-specific fine-tuning, but standard methods apply the steering signal at every generated token, incurring constant per-token perturbation that risks degrading fluency. We ask: is dense intervention necessary? We introduce Stochastic Token Steering (STS), which gates each token independently with probability $p$, and Stochastic Block Steering (SBS), which gates a leading window once per sequence; neither requires a reward model or learned gating policy. Across two model families and two behavioral tasks, steering only 50% of the tokens recovers most of the dense-steering effect while preserving fluency, and steering as few as 30% surpasses prompt-based control. The optimal steering magnitude scales inversely with the intervention ratio, revealing that SAE-mediated control is rate-limited: the behavioral outcome depends on cumulative signal dosage across a sequence.

Problem

• activation steering
• sparse autoencoders
• large language models
• behavioral control
• token-level intervention

Innovation

• Stochastic Token Steering
• Sparse Autoencoders
• Activation Steering
• Behavioral Control
• Large Language Models