Problem
Research questions and friction points this paper is trying to address.
neural networks
feature learning
monosemantic neurons
multilayer perceptrons
data efficiency
Revenge of Monosemanticity: Specialized Neurons Improve Data Efficiency in MLPs
🤖 AI Summary
研究展示了多层感知器通过发展单义特化神经元来提高数据效率，这些神经元与特定输入空间区域的预测特征对齐，形成局部低维表示。
📝 Abstract
Understanding how neural networks learn and organize features is central to understanding their behavior. Much existing theory of feature learning has focused on the emergence of a global low-dimensional predictive geometry. We show that this picture is incomplete. In regression problems with clustered data, we demonstrate that multilayer perceptrons (MLPs) naturally develop monosemantic specialized neurons: individual neurons become strongly aligned with a specific predictive feature relevant to a particular region of the input space. Rather than learning a single global low-dimensional representation, MLPs learn a collection of local low-dimensional representations that can collectively span a high-dimensional space. This specialization provably gives MLPs a data-efficiency advantage over feature-learning methods based on a global low-dimensional representation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
monosemantic specialized neurons
local low-dimensional representations
data efficiency
🔎 Similar Papers
No similar papers found.
