ELBO-T2IAlign: A Generic ELBO-Based Method for Calibrating Pixel-level Text-Image Alignment in Diffusion Models

2025-06-11

AI Summary

Diffusion models often exhibit pixel-level text-image alignment discrepancies when generating small-sized, occluded, or rare-category objects, thereby degrading performance in downstream tasks such as segmentation and editing. To address this, we propose a training-free, architecture-agnostic alignment calibration method. Our approach introduces zero-shot referring image segmentation as a proxy task for evaluating alignment quality—first of its kind—and formulates an optimization framework grounded in the evidence lower bound (ELBO), without imposing prior assumptions about alignment errors. By analyzing diffusion attention mechanisms, we enable fine-grained, pixel-level alignment modeling. Crucially, the method requires no knowledge of bias origins and is directly applicable to diverse mainstream diffusion models. Experiments demonstrate substantial improvements in localization accuracy across multiple segmentation and generation benchmarks, particularly for small-scale, occluded, and rare-category objects.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion models excel at image generation. Recent studies have shown that these models not only generate high-quality images but also encode text-image alignment information through attention maps or loss functions. This information is valuable for various downstream tasks, including segmentation, text-guided image editing, and compositional image generation. However, current methods heavily rely on the assumption of perfect text-image alignment in diffusion models, which is not the case. In this paper, we propose using zero-shot referring image segmentation as a proxy task to evaluate the pixel-level image and class-level text alignment of popular diffusion models. We conduct an in-depth analysis of pixel-text misalignment in diffusion models from the perspective of training data bias. We find that misalignment occurs in images with small sized, occluded, or rare object classes. Therefore, we propose ELBO-T2IAlign, a simple yet effective method to calibrate pixel-text alignment in diffusion models based on the evidence lower bound (ELBO) of likelihood. Our method is training-free and generic, eliminating the need to identify the specific cause of misalignment and works well across various diffusion model architectures. Extensive experiments on commonly used benchmark datasets on image segmentation and generation have verified the effectiveness of our proposed calibration approach.

Problem

• Evaluates pixel-level text-image alignment in diffusion models
• Analyzes misalignment causes like small or rare objects
• Proposes training-free ELBO-based calibration method

Innovation

• ELBO-based calibration for text-image alignment
• Training-free method for diffusion models
• Generic solution across various architectures