🤖 AI Summary
This work addresses insufficient class separability of latent representations in deep classifiers. We propose a “latent-space point collapse” mechanism: by imposing an ℓ₂ compression term on the penultimate-layer representations—jointly optimized with standard cross-entropy loss—we compel intra-class latent features to converge toward a single point. This phenomenon arises from the push-pull tension between the two objectives and is systematically identified and formalized for the first time. Our method requires no architectural modification; it introduces only a lightweight ℓ₂ regularization into standard training and employs a low-dimensional linear penultimate layer to enhance geometric interpretability. Theoretical analysis and extensive experiments demonstrate that this strategy significantly improves feature discriminability and model Lipschitz continuity, yielding concurrent gains in robustness and generalization under both clean and adversarial settings. The approach is fully plug-and-play.
📝 Abstract
The configuration of latent representations plays a critical role in determining the performance of deep neural network classifiers. In particular, the emergence of well-separated class embeddings in the latent space has been shown to improve both generalization and robustness. In this paper, we propose a method to induce the collapse of latent representations belonging to the same class into a single point, which enhances class separability in the latent space while enforcing Lipschitz continuity in the network. We demonstrate that this phenomenon, which we call extit{latent point collapse}, is achieved by adding a strong $L_2$ penalty on the penultimate-layer representations and is the result of a push-pull tension developed with the cross-entropy loss function. In addition, we show the practical utility of applying this compressing loss term to the latent representations of a low-dimensional linear penultimate layer. The proposed approach is straightforward to implement and yields substantial improvements in discriminative feature embeddings, along with remarkable gains in robustness to input perturbations.