SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation

2024-12-08

arXiv.org

AI Summary

Current multimodal large models (LMMs) face critical bottlenecks in compositional text-to-image (T2I) generation, including inaccurate text–image alignment, heavy reliance on manual annotations, and labor-intensive prompt engineering. To address these limitations, we propose a model-agnostic iterative self-improvement framework, introducing the first general-purpose self-optimization paradigm applicable to both continuous and discrete visual representations. Key innovations include: (i) kernel-based continuous direct preference optimization (DPO), which extends conventional DPO beyond discrete token spaces; (ii) a visual representation diversity enhancement mechanism; and (iii) self-feedback-driven alignment optimization. Evaluated on T2I-CompBench++ and DPG-Bench, our method achieves over 30% and approximately 20% performance gains, respectively—significantly outperforming existing layout-planning approaches and methods relying on human or AI feedback.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Multimodal Models (LMMs) have demonstrated impressive capabilities in multimodal understanding and generation, pushing forward advancements in text-to-image generation. However, achieving accurate text-image alignment for LMMs, particularly in compositional scenarios, remains challenging. Existing approaches, such as layout planning for multi-step generation and learning from human feedback or AI feedback, depend heavily on prompt engineering, costly human annotations, and continual upgrading, limiting flexibility and scalability. In this work, we introduce a model-agnostic iterative self-improvement framework (SILMM) that can enable LMMs to provide helpful and scalable self-feedback and optimize text-image alignment via Direct Preference Optimization (DPO). DPO can readily applied to LMMs that use discrete visual tokens as intermediate image representations; while it is less suitable for LMMs with continuous visual features, as obtaining generation probabilities is challenging. To adapt SILMM to LMMs with continuous features, we propose a diversity mechanism to obtain diverse representations and a kernel-based continuous DPO for alignment. Extensive experiments on three compositional text-to-image generation benchmarks validate the effectiveness and superiority of SILMM, showing improvements exceeding 30% on T2I-CompBench++ and around 20% on DPG-Bench.

Problem

• Improving text-image alignment in compositional generation scenarios
• Reducing reliance on costly human annotations and prompt engineering
• Adapting self-improvement framework for LMMs with continuous visual features

Innovation

• Model-agnostic iterative self-improvement framework
• Direct Preference Optimization for alignment
• Kernel-based continuous DPO adaptation