ELDiff: When Evidential Learning Meets Text-to-Image Diffusion

2026-06-18

AI Summary

This work addresses the degradation of semantic consistency in multi-object text-to-image generation, commonly caused by segmentation map biases and semantic overlaps in existing diffusion models. To mitigate this, the authors propose ELDiff, an evidence learning–based supervision framework that enhances object-level consistency through two key components: a pixel-wise evidence loss that reduces overconfidence in unreliable segmentation labels, and a token-level conflict loss that suppresses semantic contradictions between objects. ELDiff seamlessly integrates into standard diffusion model training pipelines without introducing additional inference overhead. Extensive experiments demonstrate that ELDiff consistently outperforms both training-based and training-free state-of-the-art methods across multiple architectures—including Stable Diffusion v1.4, v2.1, SDXL, SD v3.5, and Qwen-Image—significantly improving semantic alignment between generated images and input prompts in complex multi-object scenarios.

Abstract

In multi-object text-to-image (T2I) diffusion, ensuring semantic consistency between textual prompts and generated visual content is crucial for image synthesis. However, such consistency constraint is often underemphasized in the denoising process of diffusion models. Although token supervised diffusion models can mitigate this issue by learning object-wise consistency between the image content and object segmentation maps, it tends to suffer from the problems of segmentation map bias and semantic overlap conflict, especially when involving multiple objects. In this paper, we propose ELDiff, a new evidential learning-supervised T2I diffusion model, which leverages the advantages of uncertainty metric and conflict detection to enhance the fault tolerance of unreliable segmentation maps and suppress semantic conflicts, strengthening object-wise consistency learning. Specifically, a pixel evidence loss is proposed to restrain overconfidence in unreliable labels through evidential regularization, and a token conflict loss is designed to weaken the contradiction between semantics through optimizing a measured conflict factor. Extensive experiments show that our ELDiff outperforms existing training based and train-free based T2I diffusion models on SD v1.4, SD v2.1, SDXL, SD v3.5, and Qwen-Image, without requiring additional inference-time manipulations. Notably, ELDiff can be seamlessly extended to the existing training pipeline of T2I diffusion models. Code can be found at https://github.com/QingtaoPan/ELDiff.

Problem

• text-to-image diffusion
• semantic consistency
• segmentation map bias
• semantic overlap conflict
• multi-object generation

Innovation

• evidential learning
• text-to-image diffusion
• semantic consistency
• conflict detection
• pixel evidence loss