DataComp-VLM: Improved Open Datasets for Vision-Language Models

2026-06-26

AI Summary

This work addresses the lack of a systematic benchmark for data curation in training vision-language models (VLMs) by introducing DataComp-VLM (DC-VLM), the first data-centric experimental benchmark tailored for VLMs. DC-VLM integrates 160 datasets comprising over 6 trillion multimodal tokens, enabling comprehensive evaluation of data selection, mixing, formatting, and sampling strategies across varying model scales and data budgets, with validation on 52 downstream tasks. The study reveals that data mixing consistently outperforms filtering, and instruction-heavy mixing strategies yield superior performance at scale. Notably, an 8B-parameter model trained on the DC-VLM baseline achieves 63.6% average accuracy across 33 core tasks, surpassing the current best open-source dataset, FineVision, by 5.4 percentage points.

Abstract

Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systematic benchmarks for evaluating such curation strategies. We introduce DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve VLM training. As part of DCVLM, we collect 160 datasets spanning four data types -- image-caption pairs, multimodal interleaved documents, text-only, and instruction-tuning data -- into a corpus of 6T multimodal tokens. DCVLM allows participants to test curation strategies (filtering, mixing, formatting, sampling) across 1B-8B models and 6.25B-200B token budgets. Models are then evaluated on a carefully selected suite of up to 52 downstream benchmarks across 9 domains. We conduct extensive experiments on DCVLM and find that data mixing, not filtering, is key to a high-quality training dataset: instruction-heavy mixtures scale better than caption-heavy ones, with gains widening at larger scales. The resulting dataset, DCVLM-Baseline, enables training an 8B VLM to 63.6% accuracy on our 33-task core suite with 200B training tokens. Compared to FineVision, the state-of-the-art open VLM training dataset, this represents an improvement of +5.4pp. DCVLM and all accompanying artifacts will be made publicly available at https://www.datacomp.ai/dcvlm/.

Problem

• Vision-Language Models
• dataset curation
• benchmark
• data-centric experiments
• multimodal training data

Innovation

• DataComp-VLM
• data curation
• vision-language models
• instruction-tuning
• multimodal benchmark