DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models

2026-04-18

AI Summary

Existing vision-language datasets often suffer from noisy, redundant, and poorly aligned image–text pairs, which hinder the performance of large multimodal models. This work proposes DOSE, an efficient data curation method that requires neither fine-tuning nor task-specific training. By leveraging off-the-shelf pretrained models to jointly assess text quality and image–text alignment, DOSE constructs a joint quality–alignment distribution and selects an informative, diverse subset through adaptive weighted sampling. Experiments demonstrate that models trained on the DOSE-curated subset achieve performance on par with or superior to those trained on the full dataset across standard VQA and mathematical reasoning benchmarks, while substantially reducing data processing costs. These results validate the method’s effectiveness, efficiency, and scalability.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

High-quality and diverse multimodal data are essential for improving vision-language models (VLMs), yet existing datasets often contain noisy, redundant, and poorly aligned samples. To address these problems, data filtering is commonly used to enhance the efficiency and performance of multimodal learning, but it introduces extra computational cost because filtering models are usually trained on the same data they are meant to screen. To reduce this cost, we study DOSE, which explores whether off-the-shelf pretrained models that have never seen the target data can be used to select training samples for larger and stronger multimodal models without any task-specific training. Even without fine-tuning, these models can effectively assess text quality and image-text alignment to guide data selection. Based on this, we build a joint quality-alignment distribution and apply adaptive weighted sampling to select informative samples while maintaining long-tail diversity. This approach enhances data diversity, enabling models trained on DOSE-filtered data to match or surpass those trained on the full dataset on standard VQA and math benchmarks. Extensive experiments demonstrate its effectiveness, efficiency, and scalability.

Problem

• multimodal data
• data selection
• vision-language models
• data filtering
• off-the-shelf models

Innovation

• data selection
• off-the-shelf models
• multimodal LLMs
• quality-alignment distribution
• adaptive sampling