Mind the Performance Gap: Capability-Behavior Trade-offs in Feature Steering

2026-02-03

AI Summary

This study addresses the underexplored trade-off between behavioral control and task performance when applying feature steering to large language models. Using Goodfire’s Auto Steer method on Llama-8B and Llama-70B, the authors conduct the first systematic quantification of this tension across 14 behavioral objectives and 171 MMLU benchmark questions. While Auto Steer improves behavioral alignment—evidenced by increased control scores (e.g., from 2.98 to 3.30 on Llama-8B)—it concurrently induces a substantial drop in MMLU accuracy (from 66% to 46%) and degrades text coherence. Overall, steered models underperform simple prompt engineering baselines, raising critical concerns about the practical viability of feature steering for real-world deployment where both safety and task competence are essential.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Intelligent Robots: Behavior Learning & Control

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Feature steering has emerged as a promising approach for controlling LLM behavior through direct manipulation of internal representations, offering advantages over prompt engineering. However, its practical effectiveness in real-world applications remains poorly understood, particularly regarding potential trade-offs with output quality. We show that feature steering methods substantially degrade model performance even when successfully controlling target behaviors, a critical trade-off. Specifically, we evaluate Goodfire's Auto Steer against prompt engineering baselines across 14 steering queries (covering innocuous and safety-relevant behaviors) on 171 Massive Multitask Language Understanding (MMLU) questions using Llama-8B and Llama-70B, measuring accuracy, coherence, and behavioral control. Our findings show that Auto Steer successfully modifies target behaviors (achieving scores of 3.33 vs. 2.98 for prompting on Llama-8B and 3.57 vs. 3.10 on Llama-70B), but causes dramatic performance degradation: accuracy on the MMLU questions drops from 66% to 46% on Llama-8B and 87% to 73% on Llama-70B, with coherence falling from 4.62 to 2.24 and 4.94 to 3.89 respectively. Simple prompting achieves the best overall balance. These findings highlight limitations of current feature steering methods for practical deployment where task performance cannot be sacrificed. More broadly, our work demonstrates that mechanistic control methods face fundamental capability-behavior trade-offs that must be empirically characterized before deployment.

Problem

• feature steering
• capability-behavior trade-off
• large language models
• behavioral control
• model performance degradation

Innovation

• feature steering
• capability-behavior trade-off
• large language models
• behavioral control
• prompt engineering