Winner-Take-All bottlenecks enforce disentangled symbolic representations in multi-task learning

2026-05-21

AI Summary

This work addresses the challenge in multitask learning where deep neural networks struggle to extract interpretable, disentangled symbolic representations from highly nonlinear and entangled latent variables. The authors propose incorporating a Winner-Take-All (WTA) bottleneck into deep networks to enforce the learning of categorical abstract features, such that individual neurons or neuron groups encode specific semantic attributes—such as object identity, color, or spatial location. For the first time, they theoretically demonstrate that under certain conditions, the WTA bottleneck provably induces the spontaneous emergence of highly symbolic and disentangled internal representations. Empirical validation further confirms its effectiveness even in practical architectures that violate the theoretical assumptions. Experiments on two datasets show significant improvements in generalization, offering a novel interface between symbolic and subsymbolic artificial intelligence.

Abstract

Winner-take-all (WTA) networks constitute a central circuit motif in cortical networks of the brain. In addition, WTA-like activations are abundant in modern deep learning models in the form of the softmax activation for example in attention layers of transformers. While their role in the extraction of latent factors has been studied for relatively simple generative models, their role in the context of highly non-linearly entangled latent factors has remained elusive. In this article, we show that a WTA bottleneck within a deep neural network can enforce under certain well-defined conditions the extraction of categorical latent factors of the data in a multi-task learning setup. In particular, we prove that the representation that emerges in the WTA bottleneck is highly symbolic, where a single neuron or a population of neurons encodes the presence of a single abstract feature such as a specific object, color, or position. We furthermore show empirically on two datasets, that this also holds for architectures and setups that do not fully comply with the assumptions of our theorem and demonstrate the advantages of the acquired symbolic representation for generalization. Our proposed model provides insights into the generalization capabilities of deep neural networks with WTA-like components and may serve as an interface between symbolic and subsymbolic AI systems.

Problem

• Winner-Take-All
• disentangled representations
• symbolic representations
• multi-task learning
• latent factors

Innovation

• Winner-Take-All
• symbolic representation
• disentanglement
• multi-task learning
• neural bottleneck