Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignment

2026-06-29

AI Summary

This work addresses the susceptibility of text-to-image diffusion models to conceptual association bias when generating “one-and-only” (OAO) objects, which often leads to failures in faithfully rendering textual descriptions that contradict default visual priors. Through an information-theoretic analysis, the study reveals for the first time that critical conceptual information present in intermediate layers of the text encoder is lost in the final text embedding during propagation. To mitigate this, the authors propose a fine-tuning-free and external-model-free guidance mechanism that injects intermediate hidden states from the text encoder during early denoising steps, effectively recovering suppressed concepts. The method substantially improves counterfactual prompt alignment across four benchmarks—including OAO-AttackBench—with VQAScore gains of up to 19.1 percentage points while preserving image fidelity and human preference.

Abstract

Text-to-image (T2I) diffusion models often fail to faithfully render explicit textual descriptions, instead defaulting to strongly learned visual priors due to a phenomenon referred to as concept association bias. We show that such bias is particularly strong for one-and-only (OAO) objects, entities that exist in a single canonical form, such as celestial bodies, landmarks, and artworks. The deeply ingrained visual identity for these concepts often resists modification through prompting alone. Addressing this challenge, we first identify through an information-theoretic analysis that the final text embedding discards concept-level information present in the intermediate-layer text representations, reducing the mutual information available to the subsequent denoising process. We then propose Intermediate Text Representation (IR)-guided diffusion, which injects intermediate hidden states of the text encoder into the conditioning signal during early denoising steps, recovering suppressed concepts without any additional training, optimization, or external models. To systematically evaluate the challenging task of aligning generative outputs with unusual prompts for OAO objects, we introduce OAO-AttackBench, a benchmark comprising counterfactual prompts that directly conflict with the core visual identity of OAO objects. Experiments on four benchmarks, including OAO-AttackBench, show that our method achieves up to a 19.1 percentage-point improvement in VQAScore while preserving generation fidelity and human preference. Project page: https://soyoun-won.github.io/one-and-only-ir-guidance/.

Problem

• text-to-image generation
• concept association bias
• one-and-only objects
• visual priors
• text alignment

Innovation

• Intermediate Text Representation
• One-and-Only Objects
• Concept Association Bias
• Text-to-Image Generation
• Diffusion Models