Understanding Deep Representation Learning via Layerwise Feature Compression and Discrimination
This work investigates the fundamental mechanisms underlying hierarchical representation learning in deep neural networks. Addressing the central question—“how do features evolve across layers”—we propose a joint quantification framework for inter-layer feature compression ratio and discriminability. We theoretically uncover, for the first time, a geometric–linear dual-rate pattern of feature evolution in deep linear networks: intra-class features contract geometrically, while inter-class discriminability increases linearly. This pattern is rigorously established under minimal norm, weight balancing, and near-low-rank assumptions, and extended to nonlinear networks via intermediate-feature modeling for multi-class classification. Numerical experiments validate its robustness across architectures and datasets. Our results provide an interpretable theoretical foundation for representation learning and yield quantitative guidance for layer selection in transfer learning and knowledge distillation.