Replacement Learning: Training Neural Networks with Fewer Parameters

2026-05-19

AI Summary

This work addresses the inefficiencies of full-depth backpropagation in training deep neural networks, which suffers from parameter redundancy, high activation memory consumption, and prolonged training latency—particularly when adjacent layers exhibit highly correlated learning patterns. To mitigate these issues, the authors propose a novel paradigm termed Replacement Learning (RepL), which dynamically synthesizes proxy operators via lightweight computational layers to act on activations from preceding blocks, thereby preserving local contextual continuity while reducing redundant computation. Unlike conventional compression methods that prune or remove components, RepL innovatively replaces structural elements through a learnable parameter fusion mechanism, making it adaptable to diverse architectures such as CNNs and Vision Transformers (ViTs) and applicable across multiple tasks including classification, detection, and segmentation. Experiments on CIFAR-10, ImageNet, and COCO demonstrate that RepL substantially reduces trainable parameters, GPU memory usage, and training time while matching or surpassing standard end-to-end training performance, with strong compatibility in quantization and transfer learning scenarios.

Abstract

End-to-end training with full-depth backpropagation remains the dominant paradigm for optimizing deep neural networks, but its efficiency deteriorates as models grow deeper. Since every block must be executed and differentiated under a single global objective, full-depth BP introduces substantial parameter redundancy, activation-memory cost, and training latency, especially when neighboring layers exhibit highly correlated learning patterns. Directly skipping or removing layers can reduce cost, but often weakens representation capacity or requires architecture-specific reuse designs. In this paper, we propose Replacement Learning (RepL), a training-time paradigm that reduces full-depth redundancy by replacing selected blocks rather than simply discarding them. For each removed block, RepL inserts a lightweight computing layer that synthesizes a surrogate operator from the parameters of its adjacent preceding and succeeding blocks through a learnable transformation, and applies the synthesized operator to the preceding activation. In this way, RepL preserves local contextual continuity while avoiding unnecessary full-layer computation. We instantiate RepL for CNNs and ViTs with tailored parameter-fusion blocks that handle convolutional channels, feature resolutions, and transformer submodules. Extensive experiments on CIFAR-10, SVHN, STL-10, ImageNet, COCO, and CityScapes show that RepL reduces trainable parameters, GPU memory usage, and training time while matching or surpassing standard end-to-end training across classification, detection, and segmentation. Additional results on WikiText-2, transfer learning, inference throughput, checkpointing, stochastic depth, and INT8 quantization further demonstrate its generality and compatibility.

Problem

• parameter redundancy
• activation-memory cost
• training latency
• deep neural networks
• full-depth backpropagation

Innovation

• Replacement Learning
• parameter redundancy reduction
• lightweight surrogate operator
• full-depth backpropagation
• parameter-fusion block