A Deeper Analysis of Block-Sparse Featurizers

2026-08-27

AI Summary

本文分析了块稀疏特征提取器（BSF）的优缺点，针对其存在的特征分裂等问题，提出了一种竞赛Top-K选择规则等改进方法。

Abstract

The recently introduced block-sparse featurizer (BSF; Fel et al., 2026) is similar to a sparse autoencoder (SAE), but its atomic unit is a small subspace (a block of directions) rather than a single direction. It is designed for features that live on low-dimensional manifolds, which are especially frequent in vision. This work studies the BSF's strengths and weaknesses, finding how it still somewhat suffers from classic SAE failure modes, like feature splitting and composition. We propose several architectural changes to the BSF, including a Tournament Top-K selection rule that significantly reduces feature splitting, and we also extend the block paradigm to the crosscoder.

Problem

• Block-Sparse Featurizer
• Low-Dimensional Manifolds
• Feature Splitting
• Feature Composition

Innovation

• Block-Sparse Featurizer
• Tournament Top-K selection rule
• crosscoder