GMAIL: Generative Modality Alignment for generated Image Learning

2026-02-17

AI Summary

This work addresses the performance degradation and potential mode collapse that arise when naively mixing generated and real images in training due to their inherent modality gap. To mitigate this, the authors propose a multimodal learning framework that explicitly treats generated images as a distinct modality and aligns them with real images in the latent space via a cross-modal alignment loss, eschewing reliance on pixel-level replacement. The approach integrates generated-image fine-tuning with multimodal representation learning and is compatible with diverse vision-language model architectures. Extensive experiments demonstrate consistent performance gains across tasks including image captioning, zero-shot image retrieval and classification, and long-form caption retrieval, with notable scalability and efficacy improvements observed when applied to large models such as LLaVA.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Generative models have made it possible to synthesize highly realistic images, potentially providing an abundant data source for training machine learning models. Despite the advantages of these synthesizable data sources, the indiscriminate use of generated images as real images for training can even cause mode collapse due to modality discrepancies between real and synthetic domains. In this paper, we propose a novel framework for discriminative use of generated images, coined GMAIL, that explicitly treats generated images as a separate modality from real images. Instead of indiscriminately replacing real images with generated ones in the pixel space, our approach bridges the two distinct modalities in the same latent space through a multi-modal learning approach. To be specific, we first fine-tune a model exclusively on generated images using a cross-modality alignment loss and then employ this aligned model to further train various vision-language models with generated images. By aligning the two modalities, our approach effectively leverages the benefits of recent advances in generative models, thereby boosting the effectiveness of generated image learning across a range of vision-language tasks. Our framework can be easily incorporated with various vision-language models, and we demonstrate its efficacy throughout extensive experiments. For example, our framework significantly improves performance on image captioning, zero-shot image retrieval, zero-shot image classification, and long caption retrieval tasks. It also shows positive generated data scaling trends and notable enhancements in the captioning performance of the large multimodal model, LLaVA.

Problem

• modality discrepancy
• generated images
• vision-language models
• image learning
• synthetic data

Innovation

• Generative Modality Alignment
• Cross-modality Alignment
• Generated Image Learning
• Vision-Language Models
• Latent Space Alignment