ADAPT: Attention Driven Adaptive Prompt Scheduling and InTerpolating Orthogonal Complements for Rare Concepts Generation

2026-03-19

AI Summary

This work addresses the challenge of accurately synthesizing rare attribute combinations in text-to-image generation with diffusion models, which often struggle due to insufficient coverage in training data. The authors propose a training-free framework that integrates deterministic prompt scheduling based on attention scores with an orthogonal semantic completion interpolation mechanism. This approach effectively mitigates generation instability arising from language model stochasticity and abrupt embedding switches. By leveraging attention-guided prompt scheduling, orthogonal vector interpolation, and semantic alignment with pretrained diffusion models, the method significantly enhances the quality of generated images involving rare concept combinations. Evaluated on the RareBench benchmark, the proposed technique consistently outperforms existing methods in terms of generation accuracy, semantic consistency, and visual fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Generating rare compositional concepts in text-to-image synthesis remains a challenge for diffusion models, particularly for attributes that are uncommon in the training data. While recent approaches, such as R2F, address this challenge by utilizing LLM for prompt scheduling, they suffer from inherent variance due to the randomness of language models and suboptimal guidance from iterative text embedding switching. To address these problems, we propose the ADAPT framework, a training-free framework that deterministically plans and semantically aligns prompt schedules, providing consistent guidance to enhance the composition of rare concepts. By leveraging attention scores and orthogonal components, ADAPT significantly enhances compositional generation of rare concepts in the RareBench benchmark without additional training or fine-tuning. Through comprehensive experiments, we demonstrate that ADAPT achieves superior performance in RareBench and accurately reflects the semantic information of rare attributes, providing deterministic and precise control over the generation of rare compositions without compromising visual integrity.

Problem

• rare concepts
• text-to-image synthesis
• diffusion models
• compositional generation
• prompt scheduling

Innovation

• prompt scheduling
• orthogonal complements
• attention mechanism
• rare concept generation
• training-free framework