Stable Coresets via Posterior Sampling: Aligning Induced and Full Loss Landscapes

2025-11-21

AI Summary

To address escalating training costs induced by ever-larger deep learning models, existing gradient-based coreset selection methods suffer from performance degradation due to time-varying loss curvature mismatch and limited robustness against stochastic baselines. This paper proposes a posterior-weight-sampling-based framework for stable coreset selection. First, we establish a theoretical connection between the weight posterior distribution and the loss manifold. Second, we introduce a smoothed loss function via posterior averaging to mitigate curvature mismatch. Third, we design a gradient-driven, posterior-aware coreset selection algorithm. The framework preserves computational efficiency while significantly enhancing convergence stability and generalization performance in small-data regimes. Extensive experiments demonstrate that our method surpasses state-of-the-art approaches across multiple benchmark datasets, achieving notable generalization gains of 3.2–5.8 percentage points under high-noise conditions.

Technology Category

• Machine Learning: Learning with Manifolds
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

As deep learning models continue to scale, the growing computational demands have amplified the need for effective coreset selection techniques. Coreset selection aims to accelerate training by identifying small, representative subsets of data that approximate the performance of the full dataset. Among various approaches, gradient based methods stand out due to their strong theoretical underpinnings and practical benefits, particularly under limited data budgets. However, these methods face challenges such as naive stochastic gradient descent (SGD) acting as a surprisingly strong baseline and the breakdown of representativeness due to loss curvature mismatches over time. In this work, we propose a novel framework that addresses these limitations. First, we establish a connection between posterior sampling and loss landscapes, enabling robust coreset selection even in high data corruption scenarios. Second, we introduce a smoothed loss function based on posterior sampling onto the model weights, enhancing stability and generalization while maintaining computational efficiency. We also present a novel convergence analysis for our sampling-based coreset selection method. Finally, through extensive experiments, we demonstrate how our approach achieves faster training and enhanced generalization across diverse datasets than the current state of the art.

Problem

• Addressing computational demands in deep learning via coreset selection
• Overcoming gradient-based method limitations with posterior sampling
• Improving training efficiency and generalization across datasets

Innovation

• Using posterior sampling for robust coreset selection
• Introducing smoothed loss function for enhanced stability
• Providing convergence analysis for sampling-based coreset method