RADIANCE: Relative Adaptive Denoising with IP-Adapter for Novel Concept Enhancement

2026-07-06

AI Summary

This work addresses the challenges of concept omission and semantic drift in text-to-image diffusion models when generating rare concepts, such as unconventional attribute-object compositions. The authors propose a training-free inference framework that formulates the denoising process as a closed-loop feedback system. By integrating compositional similarity monitoring, a bidirectional scale controller, and a feedback-guided scheduler—combined with CLIP intermediate representation supervision, IP-Adapter positive-negative scale modulation, and a hierarchical alternating guidance strategy—the method dynamically balances semantic components within multi-object prompts. Evaluated on RareBench and T2I-CompBench, the approach significantly outperforms existing methods, achieving consistent improvements across key metrics including single-sample success rate, effective throughput, compositional alignment, and perceptual quality.

Abstract

Text-to-image (T2I) diffusion models have achieved striking progress but still struggle to synthesize rare concepts involving unusual attribute-object pairings, often resulting in concept omission or semantic drift where a dominant entity overwhelms the generation. Tracing these failures to a lack of compositional balance during the denoising trajectory, we propose RADIANCE, a training-free framework that treats inference as a closed-loop feedback process. RADIANCE augments pretrained backbones with three modular components: (1) a Compositional Similarity Monitor (CSM) that tracks the emergence of objects and attributes in intermediate latents via CLIP-based feedback; (2) a Bidirectional Scale Controller (BSC) that applies a reactive "restoring force" using positive and negative IP-Adapter scales to rebalance biased trajectories; and (3) a Feedback Guidance Scheduler (FGS) that coordinates these updates across timesteps without additional training. We further extend the framework to multi-object prompts via Delayed Adapter Activation (DAA) and Layer-wise Alternating Guidance (LAG) to prevent premature concept fusion. By overlapping monitoring and denoising through pipelined execution, RADIANCE maintains competitive latency while significantly enhancing the per-sample success rate and effective throughput. Experiments on RareBench and T2I-CompBench demonstrate that RADIANCE consistently enhances compositional alignment and perceptual quality over state-of-the-art baselines.

Problem

• text-to-image generation
• rare concepts
• compositional balance
• semantic drift
• concept omission

Innovation

• training-free
• compositional balance
• IP-Adapter
• closed-loop feedback
• denoising trajectory