🤖 AI Summary
Existing synthetic data generation methods primarily focus on scaling data volume, yet suffer from limited semantic diversity and coarse-grained image-text alignment, leading to redundancy and superficial descriptions. To address this, we propose Role-SynthCLIP, a novel framework that introduces a role-playing mechanism—e.g., “Composition Analyst” and “Context Interpreter”—to guide multimodal large language models (MLLMs) in generating semantically rich, fine-grained aligned image-text pairs from diverse perspectives, without increasing dataset size. This enhances both semantic diversity and cross-modal alignment precision. The high-quality synthetic data produced by Role-SynthCLIP is employed for contrastive pretraining of CLIP. Using only 1 million synthetic pairs, our method achieves 64.1% Recall@1 on MS COCO—outperforming the 5-million-pair baseline by 2.8 percentage points. These results empirically validate the efficacy of role-driven prompting in improving synthetic data quality and downstream representation learning.
📝 Abstract
The effectiveness of Contrastive Language-Image Pre-training (CLIP) models critically depends on the semantic diversity and quality of their training data. However, while existing synthetic data generation methods primarily focus on increasing data volume, such emphasis often leads to limited semantic diversity and redundant or shallow captions. To address this limitation, we propose Role-SynthCLIP, a novel data synthesis framework that leverages multi-perspective role-playing prompts (e.g., a compositional analyst, an interpreter of image context) to guide Multimodal Large Language Models (MLLMs) in generating semantically diverse captions from distinct viewpoints. This mechanism enhances the semantic diversity and fine-grained image-text alignment of synthetic pairs, thereby improving caption expressiveness and accuracy while keeping the total number of image-text pairs unchanged. Experimental results demonstrate the effectiveness and efficiency of our method. A CLIP-B/16 model trained on only 1 million Role-SynthCLIP pairs achieves a Recall@1 of 64.1% on the MS COCO validation set, surpassing the best existing synthetic data baseline (trained on 5M pairs) by 2.8 percentage points. The code and trained models are released at https://github.com/huangfu170/Role-SynthCLIP.