Efficient Learned Image Compression Through Knowledge Distillation

2025-09-12

AI Summary

To address the high computational overhead and poor real-time deployability of neural image compression models on resource-constrained devices, this paper introduces, for the first time, a systematic knowledge distillation framework tailored to image compression. The proposed lightweight distillation paradigm leverages a high-performance teacher model to guide the training of a compact student model, integrating entropy coding optimization, latent-space quantization, and multi-scale feature mapping. Compared with conventional approaches, our method achieves near-teacher rate-distortion performance while substantially reducing model parameters (−72% on average), inference latency (+3.1× speedup), and energy consumption (−58%). It demonstrates robust generalization across diverse rate-quality operating points. Experimental results show consistent superiority over state-of-the-art lightweight compression models under multi-scale architectures.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Learning & Optimization for CV
• Data Mining & Knowledge Management: Data Compression

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Learned image compression sits at the intersection of machine learning and image processing. With advances in deep learning, neural network-based compression methods have emerged. In this process, an encoder maps the image to a low-dimensional latent space, which is then quantized, entropy-coded into a binary bitstream, and transmitted to the receiver. At the receiver end, the bitstream is entropy-decoded, and a decoder reconstructs an approximation of the original image. Recent research suggests that these models consistently outperform conventional codecs. However, they require significant processing power, making them unsuitable for real-time use on resource-constrained platforms, which hinders their deployment in mainstream applications. This study aims to reduce the resource requirements of neural networks used for image compression by leveraging knowledge distillation, a training paradigm where smaller neural networks, partially trained on the outputs of larger, more complex models, can achieve better performance than when trained independently. Our work demonstrates that knowledge distillation can be effectively applied to image compression tasks: i) across various architecture sizes, ii) to achieve different image quality/bit rate tradeoffs, and iii) to save processing and energy resources. This approach introduces new settings and hyperparameters, and future research could explore the impact of different teacher models, as well as alternative loss functions. Knowledge distillation could also be extended to transformer-based models. The code is publicly available at: https://github.com/FABallemand/PRIM .

Problem

• Reducing neural network resource requirements for image compression
• Applying knowledge distillation to improve efficiency across architectures
• Enabling real-time deployment on resource-constrained platforms

Innovation

• Knowledge distillation reduces neural network size
• Smaller networks achieve better compression performance
• Saves processing and energy resources effectively