Diffusion Reconstruction-based Data Likelihood Estimation for Core-Set Selection

2025-11-24

AI Summary

Existing core-set selection methods predominantly rely on heuristic scoring (e.g., training dynamics, uncertainty) without explicit modeling of data likelihood, limiting their ability to capture essential distributional structure. Method: This paper introduces, for the first time, the partial denoising reconstruction bias of diffusion models into core-set selection. We theoretically establish that this bias is negatively correlated with the log-likelihood of data and rigorously link it to the evidence lower bound (ELBO). Furthermore, we propose an information-entropy-driven, timestep-adaptive selection mechanism to enable distribution-aware, precise data scoring. Results: Evaluated on ImageNet, our method achieves full-dataset performance using only 50% of the data—outperforming state-of-the-art approaches by a significant margin—while demonstrating robustness and stability across varying subset sizes.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Dimensionality Reduction/Feature Selection
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Existing core-set selection methods predominantly rely on heuristic scoring signals such as training dynamics or model uncertainty, lacking explicit modeling of data likelihood. This omission may hinder the constructed subset from capturing subtle yet critical distributional structures that underpin effective model training. In this work, we propose a novel, theoretically grounded approach that leverages diffusion models to estimate data likelihood via reconstruction deviation induced by partial reverse denoising. Specifically, we establish a formal connection between reconstruction error and data likelihood, grounded in the Evidence Lower Bound (ELBO) of Markovian diffusion processes, thereby enabling a principled, distribution-aware scoring criterion for data selection. Complementarily, we introduce an efficient information-theoretic method to identify the optimal reconstruction timestep, ensuring that the deviation provides a reliable signal indicative of underlying data likelihood. Extensive experiments on ImageNet demonstrate that reconstruction deviation offers an effective scoring criterion, consistently outperforming existing baselines across selection ratios, and closely matching full-data training using only 50% of the data. Further analysis shows that the likelihood-informed nature of our score reveals informative insights in data selection, shedding light on the interplay between data distributional characteristics and model learning preferences.

Problem

• Estimating data likelihood via diffusion reconstruction for core-set selection
• Addressing lack of explicit data modeling in existing selection methods
• Developing distribution-aware scoring using reconstruction deviation and ELBO

Innovation

• Leverages diffusion models for data likelihood estimation
• Connects reconstruction error to data likelihood via ELBO
• Uses information-theoretic method for optimal reconstruction timestep