Compressing image encoders via latent distillation

2026-01-09

arXiv.org

AI Summary

This work addresses the challenge of deploying deep image compression models in resource-constrained settings, where existing approaches suffer from encoder complexity and high computational costs. To overcome this, the authors propose an encoder compression method based on latent-space knowledge distillation, combining a simplified distillation strategy with a lightweight network architecture. This approach significantly reduces model complexity while effectively approximating the latent representations of the original model. Notably, it drastically lowers both training data requirements and training time. Experiments on two mainstream architectures demonstrate its superiority over directly training lightweight encoders, achieving better reconstruction quality and statistical fidelity. The method thus offers a practical solution for efficient deployment of deep image compression systems.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Computer Vision: Diffusion Models for Vision
• Data Mining & Knowledge Management: Data Compression

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Deep learning models for image compression often face practical limitations in hardware-constrained applications. Although these models achieve high-quality reconstructions, they are typically complex, heavyweight, and require substantial training data and computational resources. We propose a methodology to partially compress these networks by reducing the size of their encoders. Our approach uses a simplified knowledge distillation strategy to approximate the latent space of the original models with less data and shorter training, yielding lightweight encoders from heavyweight ones. We evaluate the resulting lightweight encoders across two different architectures on the image compression task. Experiments show that our method preserves reconstruction quality and statistical fidelity better than training lightweight encoders with the original loss, making it practical for resource-limited environments.

Problem

• image compression
• hardware-constrained applications
• heavyweight models
• computational resources
• training data

Innovation

• latent distillation
• image compression
• knowledge distillation
• lightweight encoder
• neural network compression