Data Selection Through Iterative Self-Filtering for Vision-Language Settings

2026-06-22

AI Summary

This work addresses the challenge of pervasive noise in large-scale vision–language datasets, where manual curation is prohibitively expensive and impractical. The authors propose a bootstrapped, iterative self-filtering approach that operates without reliance on external clean data or pretrained models. During CLIP training, the method dynamically evaluates and selects high-quality, diverse samples by alternately optimizing model parameters and data mixture strategies, thereby achieving an effective balance between data purification and diversity preservation. Extensive experiments demonstrate that this approach significantly enhances model performance across multiple downstream tasks, underscoring its efficacy and generalizability.

Abstract

The availability of large amounts of clean data is paramount to training neural networks. However, at large scales, manual oversight is impractical, resulting in sizeable datasets that can be very noisy. Attempts to mitigate this obstacle to producing performant vision-language models have so far involved heuristics, curated reference datasets, and using pre-trained models. Here we propose a novel, bootstrapped method in which a CLIP model is trained on an evolving, self-selected dataset. This evolving dataset constitutes a balance of filtered, highly probable clean samples as well as diverse samples from the entire distribution. Our proposed Self-Filtering method iterates between training the model and selecting a subsequently improved data mixture. Training on vision-language datasets filtered by the proposed approach improves downstream performance without the need for additional data or pre-trained models.

Problem

• data selection
• vision-language models
• noisy data
• dataset filtering
• clean data

Innovation

• Self-Filtering
• vision-language models
• data selection
• iterative training
• CLIP