A Novel Structure-Agnostic Multi-Objective Approach for Weight-Sharing Compression in Deep Neural Networks

2025-01-06

AI Summary

To address the high weight storage overhead and deployment challenges of deep neural networks on embedded devices, this paper proposes a structural-, task-, and dataset-agnostic weight-sharing compression framework. The method employs a multi-objective evolutionary algorithm (MOEA)-driven unified interval quantization scheme, where cluster centroids directly serve as shared weights—eliminating the need for post-quantization fine-tuning. It further introduces iterative neighboring bin merging and Pareto-front optimization to achieve optimal trade-offs between accuracy and compression ratio. With only *O*(*N*) clustering complexity, the framework attains 13.72–14.98×, 11.61–12.99×, and 7.44–8.58× memory compression on CIFAR-10, CIFAR-100, and ImageNet, respectively, while maintaining controlled accuracy degradation. Key contributions include: (1) a retraining-free, centroid-based quantization strategy; and (2) an efficient, principled optimization framework balancing fidelity and compactness across diverse architectures and tasks.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Learning & Optimization for CV
• Data Mining & Knowledge Management: Data Compression

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Deep neural networks suffer from storing millions and billions of weights in memory post-training, making challenging memory-intensive models to deploy on embedded devices. The weight-sharing technique is one of the popular compression approaches that use fewer weight values and share across specific connections in the network. In this paper, we propose a multi-objective evolutionary algorithm (MOEA) based compression framework independent of neural network architecture, dimension, task, and dataset. We use uniformly sized bins to quantize network weights into a single codebook (lookup table) for efficient weight representation. Using MOEA, we search for Pareto optimal $k$ bins by optimizing two objectives. Then, we apply the iterative merge technique to non-dominated Pareto frontier solutions by combining neighboring bins without degrading performance to decrease the number of bins and increase the compression ratio. Our approach is model- and layer-independent, meaning the weights are mixed in the clusters from any layer, and the uniform quantization method used in this work has $O(N)$ complexity instead of non-uniform quantization methods such as k-means with $O(Nkt)$ complexity. In addition, we use the center of clusters as the shared weight values instead of retraining shared weights, which is computationally expensive. The advantage of using evolutionary multi-objective optimization is that it can obtain non-dominated Pareto frontier solutions with respect to performance and shared weights. The experimental results show that we can reduce the neural network memory by $13.72 sim14.98 imes$ on CIFAR-10, $11.61 sim 12.99 imes$ on CIFAR-100, and $7.44 sim 8.58 imes$ on ImageNet showcasing the effectiveness of the proposed deep neural network compression framework.

Problem

• Deep Neural Networks
• Memory Efficiency
• Model Deployment

Innovation

• Evolutionary Algorithm
• Neural Network Compression
• Weight Sharing