Distilling Dataset into Neural Field

2025-03-05

AI Summary

To address the high computational and memory overhead of training on large-scale datasets, this paper proposes a data distillation framework based on Implicit Neural Representations (INRs). Unlike conventional explicit sample-based distillation, our approach parameterizes the entire dataset as a lightweight neural field, implicitly encoding essential training information via coordinate-to-feature mapping—enabling, for the first time, unified distillation across modalities (images, videos, audio, and 3D voxels). We theoretically establish that, under identical parameter budgets, our representation exhibits superior expressive power compared to existing methods. Experiments on multimodal benchmarks demonstrate that synthesized data volumes can be reduced by over 99% while preserving model training accuracy comparable to that achieved with full datasets—significantly outperforming state-of-the-art distillation approaches. Our code and models are publicly available.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Generation

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Utilizing a large-scale dataset is essential for training high-performance deep learning models, but it also comes with substantial computation and storage costs. To overcome these challenges, dataset distillation has emerged as a promising solution by compressing the large-scale dataset into a smaller synthetic dataset that retains the essential information needed for training. This paper proposes a novel parameterization framework for dataset distillation, coined Distilling Dataset into Neural Field (DDiF), which leverages the neural field to store the necessary information of the large-scale dataset. Due to the unique nature of the neural field, which takes coordinates as input and output quantity, DDiF effectively preserves the information and easily generates various shapes of data. We theoretically confirm that DDiF exhibits greater expressiveness than some previous literature when the utilized budget for a single synthetic instance is the same. Through extensive experiments, we demonstrate that DDiF achieves superior performance on several benchmark datasets, extending beyond the image domain to include video, audio, and 3D voxel. We release the code at https://github.com/aailab-kaist/DDiF.

Problem

• Reduces computation and storage costs in deep learning.
• Compresses large datasets into smaller synthetic datasets.
• Enhances dataset distillation using neural fields.

Innovation

• Neural field for dataset compression
• Coordinates-based information preservation
• Superior performance across multiple domains