Is Your Text-to-Image Model Robust to Caption Noise?

2024-12-27

AI Summary

This work addresses the insufficient robustness of text-to-image (T2I) models when conditioned on hallucinated captions generated by vision-language models (VLMs), systematically revealing the severe degradation in generated representations induced by caption quality deterioration—particularly hallucination noise. We first identify VLM output confidence as a reliable indicator of caption hallucination and, based on this insight, construct the first benchmark dataset annotated with hallucination labels. To mitigate hallucination sensitivity, we propose a confidence-aware weighted robust fine-tuning paradigm. Evaluated across multiple T2I architectures, our method improves generation consistency by 18.7% and substantially reduces hallucination susceptibility. This work provides both a novel methodology and empirical foundation for enhancing the reliability of T2I models under realistic weakly supervised settings where caption quality is uncontrolled.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

In text-to-image (T2I) generation, a prevalent training technique involves utilizing Vision Language Models (VLMs) for image re-captioning. Even though VLMs are known to exhibit hallucination, generating descriptive content that deviates from the visual reality, the ramifications of such caption hallucinations on T2I generation performance remain under-explored. Through our empirical investigation, we first establish a comprehensive dataset comprising VLM-generated captions, and then systematically analyze how caption hallucination influences generation outcomes. Our findings reveal that (1) the disparities in caption quality persistently impact model outputs during fine-tuning. (2) VLMs confidence scores serve as reliable indicators for detecting and characterizing noise-related patterns in the data distribution. (3) even subtle variations in caption fidelity have significant effects on the quality of learned representations. These findings collectively emphasize the profound impact of caption quality on model performance and highlight the need for more sophisticated robust training algorithm in T2I. In response to these observations, we propose a approach leveraging VLM confidence score to mitigate caption noise, thereby enhancing the robustness of T2I models against hallucination in caption.

Problem

• Robustness
• Visual Language Models
• Image Synthesis

Innovation

• Visual Language Models
• Confidence Scores
• Robustness Enhancement