Big2Small: A Unifying Neural Network Framework for Model Compression

2026-03-31

AI Summary

This work addresses the long-standing absence of a unified theoretical framework for model compression by formulating a cohesive mathematical model grounded in measure theory. It unifies prominent compression techniques—including pruning, quantization, low-rank decomposition, and knowledge distillation—as instances of regularized neural network optimization problems. The authors propose Big2Small, a data-agnostic compression paradigm that leverages implicit neural representations (INRs) to encode the weights of large models, enabling efficient compression and high-fidelity reconstruction. To further enhance reconstruction quality, they introduce outlier-aware preprocessing and a frequency-aware loss function. Evaluated on image classification and segmentation benchmarks, the method achieves state-of-the-art compression ratios while matching or exceeding the accuracy of existing approaches.

Abstract

With the development of foundational models, model compression has become a critical requirement. Various model compression approaches have been proposed such as low-rank decomposition, pruning, quantization, ergodic dynamic systems, and knowledge distillation, which are based on different heuristics. To elevate the field from fragmentation to a principled discipline, we construct a unifying mathematical framework for model compression grounded in measure theory. We further demonstrate that each model compression technique is mathematically equivalent to a neural network subject to a regularization. Building upon this mathematical and structural equivalence, we propose an experimentally-verified data-free model compression framework, termed \textit{Big2Small}, which translates Implicit Neural Representations (INRs) from data domain to the domain of network parameters. \textit{Big2Small} trains compact INRs to encode the weights of larger models and reconstruct the weights during inference. To enhance reconstruction fidelity, we introduce Outlier-Aware Preprocessing to handle extreme weight values and a Frequency-Aware Loss function to preserve high-frequency details. Experiments on image classification and segmentation demonstrate that \textit{Big2Small} achieves competitive accuracy and compression ratios compared to state-of-the-art baselines.

Problem

• model compression
• unifying framework
• measure theory
• neural networks
• foundational models

Innovation

• model compression
• Implicit Neural Representations
• measure theory
• data-free compression
• weight reconstruction