SteerRM: Debiasing Reward Models via Sparse Autoencoders

2026-03-13

AI Summary

Reward models are susceptible to superficial stylistic cues, often favoring responses with preferred formatting over those with superior semantic content, thereby introducing alignment bias. This work proposes an inference-time intervention that requires neither retraining nor architectural modification: leveraging sparse autoencoders (SAEs) to identify and suppress features associated with such stylistic biases. The study reveals that these bias-related features are concentrated in the shallow layers of the model and exhibit cross-architectural transferability, suggesting a common underlying encoding pattern for stylistic bias. Evaluated on RM-Bench, the method improves Hard-split accuracy by an average of 7.3 points while preserving overall performance. Furthermore, it demonstrates strong generalization across diverse settings, including Gemma-based reward models and scenarios involving non-stylistic biases.

Technology Category

• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Learning Human Values and Preferences

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Reward models (RMs) are critical components of alignment pipelines, yet they exhibit biases toward superficial stylistic cues, preferring better-presented responses over semantically superior ones. Existing debiasing methods typically require retraining or architectural modifications, while direct activation suppression degrades performance due to representation entanglement. We propose SteerRM, the first training-free method for debiasing reward models using Sparse Autoencoder (SAE)-based interventions. SteerRM isolates stylistic effects using contrastive paired responses, identifies bias-related SAE features with a strength-stability criterion, and suppresses them at inference time. Across six reward models on RM-Bench, SteerRM improves Hard-split accuracy by 7.3 points on average while preserving overall performance. Results on a Gemma-based reward model and a controlled non-format bias further suggest generalization across RM architectures and bias types. We further find that format-related features are concentrated in shallow layers and transfer across models, revealing shared architecture-level bias encoding patterns. These results show that SAE-based interventions can mitigate reward-model biases without retraining, providing a practical and interpretable solution for alignment pipelines.

Problem

• reward models
• bias
• debiasing
• alignment
• stylistic cues

Innovation

• Sparse Autoencoder
• Reward Model Debiasing
• Training-Free Intervention
• Feature Suppression
• Alignment