Finding the Muses: Identifying Coresets through Loss Trajectories

2025-03-12

AI Summary

To address scalability challenges of deep learning models under resource constraints, this paper proposes an efficient coreset selection method based on loss trajectory alignment. The core innovation is the Loss Trajectory Correlation (LTC) metric—a novel measure quantifying the dynamic relationship between individual training samples and the validation loss evolution during training—thereby transforming coreset construction into a lightweight byproduct computation task. Crucially, LTC requires no additional gradient computations or model retraining. It achieves, for the first time, robust cross-architecture transferability (across ResNet, VGG, DenseNet, and Swin) and enables interpretable analysis of training dynamics. On CIFAR-100 and ImageNet-1K, models trained on subsets selected by LTC match or surpass state-of-the-art methods in accuracy (within <1% gap), exhibit minimal architecture generalization degradation (<2%), and incur substantially lower computational overhead.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Deep learning models achieve state-of-the-art performance across domains but face scalability challenges in real-time or resource-constrained scenarios. To address this, we propose Loss Trajectory Correlation (LTC), a novel metric for coreset selection that identifies critical training samples driving generalization. $LTC$ quantifies the alignment between training sample loss trajectories and validation set loss trajectories, enabling the construction of compact, representative subsets. Unlike traditional methods with computational and storage overheads that are infeasible to scale to large datasets, $LTC$ achieves superior efficiency as it can be computed as a byproduct of training. Our results on CIFAR-100 and ImageNet-1k show that $LTC$ consistently achieves accuracy on par with or surpassing state-of-the-art coreset selection methods, with any differences remaining under 1%. LTC also effectively transfers across various architectures, including ResNet, VGG, DenseNet, and Swin Transformer, with minimal performance degradation (<2%). Additionally, LTC offers insights into training dynamics, such as identifying aligned and conflicting sample behaviors, at a fraction of the computational cost of traditional methods. This framework paves the way for scalable coreset selection and efficient dataset optimization.

Problem

• Identifies critical training samples for generalization
• Reduces computational and storage overheads in coreset selection
• Enables scalable and efficient dataset optimization

Innovation

• Loss Trajectory Correlation for coreset selection
• Efficient computation as a training byproduct
• Scalable across architectures with minimal degradation