🤖 AI Summary
Fully connected (FC) layers suffer from high memory and computational overhead due to parameter redundancy. Existing compression methods often sacrifice accuracy or introduce engineering complexity. This paper proposes the Parameterized Random Projection (PRP) layer: it employs a fixed random projection matrix for efficient feature mixing, augmented by lightweight, element-wise learnable scaling and bias modulation—thereby decoupling linear transformation from adaptive modeling. PRP is the first approach to unify fixed random projection with fine-grained, element-level modulation, reducing parameter complexity to linear while preserving strong generalization. Theoretical analysis, supported by low-rank approximation modeling, substantiates its efficacy. Experiments across multiple benchmarks demonstrate zero accuracy degradation, significant inference speedup, and substantial memory reduction—enabling deployment on resource-constrained edge devices.
📝 Abstract
Fully connected layers are a primary source of memory and computational overhead in deep neural networks due to their dense, often redundant parameterization. While various compression techniques exist, they frequently introduce complex engineering trade-offs or degrade model performance. We propose the Parametrized Random Projection (PRP) layer, a novel approach that decouples feature mixing from adaptation by utilizing a fixed random matrix modulated by lightweight, learnable element-wise parameters. This architecture drastically reduces the trainable parameter count to a linear scale while retaining reliable accuracy across various benchmarks. The design serves as a stable, computationally efficient solution for architectural scaling and deployment in resource-limited settings.