DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception

2024-05-24

arXiv.org

AI Summary

Large language model (LLM)-driven multimodal models suffer from perceptual errors and hallucinations on out-of-distribution images—particularly regarding fine-grained visual attributes such as orientation, quantity, color, and structural layout—largely due to excessive compression and loss of non-semantic yet discriminative details in the image encoder. Method: This work pioneers the use of diffusion models as a “visual perception eye” for LMMs, leveraging their generative feedback to calibrate the semantic representation distribution of the image encoder—without introducing new modules or requiring large-scale retraining. Our approach comprises three core components: a generative feedback mechanism, encoder semantic alignment, and a lightweight fine-tuning framework. Contribution/Results: Our method achieves +4.0%, +6.5%, and +12.8% improvements on RobustVQA, MMVP, and POPE benchmarks, respectively. It reduces parameter count, cuts pretraining data requirements by 90%, enables smaller base models, and significantly surpasses state-of-the-art methods in robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The development of large language models (LLMs) has significantly advanced the emergence of large multimodal models (LMMs). While LMMs have achieved tremendous success by promoting the synergy between multimodal comprehension and creation, they often face challenges when confronted with out-of-distribution data, such as which can hardly distinguish orientation, quantity, color, structure, etc. This is primarily due to their reliance on image encoders trained to encode images into task-relevant features, which may lead them to disregard irrelevant details. Delving into the modeling capabilities of diffusion models for images naturally prompts the question: Can diffusion models serve as the eyes of large language models for image perception? In this paper, we propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. This addresses the drawbacks of previous methods that solely relied on image encoders like CLIP-ViT, thereby enhancing the model's resilience against out-of-distribution samples and reducing visual hallucinations. Importantly, this is achieved without requiring additional training modules and with fewer training parameters. We extensively evaluated DEEM on both our newly constructed RobustVQA benchmark and other well-known benchmarks, POPE and MMVP, for visual hallucination and perception. In particular, DEEM improves LMM's visual perception performance to a large extent (e.g., 4% higher on RobustVQA, 6.5% higher on MMVP and 12.8 % higher on POPE ). Compared to the state-of-the-art interleaved content generation models, DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data (10%), and a smaller base model size.

Problem

• Enhances image perception in large multimodal models using diffusion models.
• Addresses challenges with out-of-distribution data in visual tasks.
• Reduces visual hallucinations without additional training modules.

Innovation

• Uses diffusion models for image perception alignment.
• Enhances model resilience with fewer training parameters.
• Improves visual perception without additional training modules.