Pruning-based Data Selection and Network Fusion for Efficient Deep Learning

2025-01-02

AI Summary

To address the low training efficiency and high annotation cost in deep learning, this paper proposes PruneFuse: a novel data selection framework. It first constructs a lightweight proxy model via structured pruning to efficiently evaluate sample-wise gradient sensitivity and identify high-information samples. Subsequently, it introduces a parameter-space network fusion mechanism—comprising weight interpolation and feature alignment—to transfer the proxy’s data selection knowledge back to the original dense model. PruneFuse is the first method to synergistically integrate model pruning and network fusion for data selection, establishing a closed-loop “proxy-guided selection → knowledge migration” paradigm that balances selection efficiency and generalization performance. Experiments on multiple benchmark datasets demonstrate that PruneFuse reduces data selection computational overhead by over 60%, accelerates end-to-end training by 1.8×, and improves final accuracy by 1.2–2.4% over state-of-the-art data selection methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Dimensionality Reduction/Feature Selection
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Efficient data selection is essential for improving the training efficiency of deep neural networks and reducing the associated annotation costs. However, traditional methods tend to be computationally expensive, limiting their scalability and real-world applicability. We introduce PruneFuse, a novel method that combines pruning and network fusion to enhance data selection and accelerate network training. In PruneFuse, the original dense network is pruned to generate a smaller surrogate model that efficiently selects the most informative samples from the dataset. Once this iterative data selection selects sufficient samples, the insights learned from the pruned model are seamlessly integrated with the dense model through network fusion, providing an optimized initialization that accelerates training. Extensive experimentation on various datasets demonstrates that PruneFuse significantly reduces computational costs for data selection, achieves better performance than baselines, and accelerates the overall training process.

Problem

• Efficient Deep Learning
• Selective Training Data
• Optimized Network Structure

Innovation

• PruneFuse
• Neural Network Pruning
• Data Selection Optimization