C$^{2}$R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders

2026-06-29

AI Summary

This work addresses the feature splitting and feature absorption phenomena commonly observed in sparse autoencoders when trained with large-scale dictionaries, which undermine the atomicity and interpretability of latent representations. To mitigate these issues, the authors propose Cross-sample Consistency Regularization (C²R), a novel regularization technique that encourages semantically similar samples within a batch to activate consistent latent units while suppressing the co-activation of latent variables with similar directions. This approach systematically enhances the discreteness and semantic clarity of learned features without compromising reconstruction fidelity. Experimental results demonstrate that C²R significantly improves the atomicity and interpretability of latent representations, offering a promising direction for advancing sparse representation learning.

Abstract

Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features, but scaling to large dictionaries exposes fundamental challenges. Systematic studies reveal pervasive feature splitting that fragments coherent concepts into non-atomic latents and widespread feature absorption that creates arbitrary exceptions in general features, severely compromising latent reliability. These issues stem from inconsistent latent assignment across samples: without cross-sample constraints, per-sample optimization often allows a single underlying concept to be inconsistently distributed across multiple redundant or interfering latents. To address this, we introduce C$^2$R (\underline{\textbf{C}}ross-sample \underline{\textbf{C}}onsistency \underline{\textbf{R}}egularization). C$^2$R explicitly encourages that each semantic feature is consistently represented by a unified latent across the batch by penalizing the co-activation of directionally similar latents. Comprehensive evaluation demonstrates that C$^2$R effectively mitigates both splitting and absorption while, crucially, preserving reconstruction fidelity, providing a principled solution that enhances latent interpretability without degrading model performance. Source code is available at https://github.com/hr-jin/Cross-sample-Consistency-Regularization.

Problem

• Sparse Autoencoders
• Feature Splitting
• Feature Absorption
• Latent Consistency
• Interpretability

Innovation

• Cross-sample Consistency Regularization
• Sparse Autoencoders
• Feature Splitting
• Feature Absorption
• Latent Interpretability