Learn&Drop: Fast Learning of CNNs based on Layer Dropping

2026-04-25

AI Summary

This work addresses the high computational cost of forward propagation during the training of deep convolutional neural networks, which significantly limits training efficiency. The authors propose a dynamic layer pruning method tailored specifically for the training phase, which continuously evaluates each layer’s parameter dynamics and learning potential to identify and prune low-contribution layers in real time. Unlike prior approaches focused on inference acceleration or backward-pass optimization, this method pioneers online forward-path compression during training, leveraging a layer-scoring mechanism to enable dynamic network scaling. Experiments on VGG and ResNet architectures across MNIST, CIFAR-10, and Imagenette datasets demonstrate over 50% reduction in training time and 17.83%–83.74% fewer forward FLOPs, all without noticeable degradation in model accuracy.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

This paper proposes a new method to improve the training efficiency of deep convolutional neural networks. During training, the method evaluates scores to measure how much each layer's parameters change and whether the layer will continue learning or not. Based on these scores, the network is scaled down such that the number of parameters to be learned is reduced, yielding a speed up in training. Unlike state-of-the-art methods that try to compress the network to be used in the inference phase or to limit the number of operations performed in the backpropagation phase, the proposed method is novel in that it focuses on reducing the number of operations performed by the network in the forward propagation during training. The proposed training strategy has been validated on two widely used architecture families: VGG and ResNet. Experiments on MNIST, CIFAR-10 and Imagenette show that, with the proposed method, the training time of the models is more than halved without significantly impacting accuracy. The FLOPs reduction in the forward propagation during training ranges from 17.83\% for VGG-11 to 83.74\% for ResNet-152. These results demonstrate the effectiveness of the proposed technique in speeding up learning of CNNs. The technique will be especially useful in applications where fine-tuning or online training of convolutional models is required, for instance because data arrive sequentially.

Problem

• training efficiency
• convolutional neural networks
• forward propagation
• layer dropping
• training acceleration

Innovation

• layer dropping
• training acceleration
• forward propagation optimization
• dynamic network pruning
• CNN training efficiency