A Study in Dataset Distillation for Image Super-Resolution

2025-02-05

AI Summary

This work pioneers the systematic application of dataset distillation to image super-resolution (SR), addressing the challenge of reducing massive training datasets while preserving model performance and training efficiency. We propose a dual-path distillation framework operating simultaneously in pixel and latent spaces: pixel-space distillation employs gradient matching, while latent-space distillation integrates a variational autoencoder (VAE) as a structural prior, augmented by a multi-stage optimization strategy. Experiments demonstrate that, using only 8.88% of the original training samples (91.12% compression rate), lightweight SR models—such as EDSR variants—achieve PSNR and SSIM scores comparable to those trained on the full dataset, alongside substantial reductions in GPU memory consumption and training time. Crucially, this study extends dataset distillation beyond its traditional confinement to classification tasks, establishing a novel data-efficient learning paradigm for generative vision tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Dataset distillation is the concept of condensing large datasets into smaller but highly representative synthetic samples. While previous research has primarily focused on image classification, its application to image Super-Resolution (SR) remains underexplored. This exploratory work studies multiple dataset distillation techniques applied to SR, including pixel- and latent-space approaches under different aspects. Our experiments demonstrate that a 91.12% dataset size reduction can be achieved while maintaining comparable SR performance to the full dataset. We further analyze initialization strategies and distillation methods to optimize memory efficiency and computational costs. Our findings provide new insights into dataset distillation for SR and set the stage for future advancements.

Problem

• Explores dataset distillation in image super-resolution
• Compares pixel- and latent-space distillation techniques
• Achieves 91.12% dataset size reduction efficiently

Innovation

• Dataset distillation for super-resolution
• Pixel- and latent-space approaches
• Memory efficiency and computational optimization