Redirecting the Flow: Image Customization through Attention Distribution Shift

2026-06-15

AI Summary

Existing image customization methods face limitations in efficiency, alignment of reference features, and suppression of irrelevant information. This work formalizes image customization as a reference-induced attention distribution shift problem and introduces a conditional attention shift framework grounded in maximum entropy theory. The proposed approach features a dual-branch architecture—Reference-Alignment and Cross-Guidance—that enables synergistic guidance from both text prompts and reference images while achieving cross-layer feature alignment. Built upon Stable Diffusion 3, the method outperforms state-of-the-art techniques on the DreamBooth and Custom101 benchmarks, achieving a superior balance between subject consistency and semantic fidelity.

Abstract

Subject-driven image customization aims to generate images that not only follow textual instructions but also preserve the identity of a given reference subject. Existing approaches, including test-time fine-tuning, encoder-based methods, and token competition in shared attention spaces, suffer from limited efficiency, misalignment between extracted reference features and the generative process, and interference from irrelevant information. To address these limitations, we formulate the customization task as a distribution shift induced by incorporating reference images into text-to-image generation, and derive a Conditional Attention Distribution Shift formulation grounded in maximum entropy theory. Building on this formulation, we propose CustomShift, a dual-branch architecture based on Stable Diffusion 3. The Reference-Alignment Branch leverages self-attention between reference images and subject names to achieve layer-wise alignment with latent representations, while the Cross-Guidance Branch integrates textual and reference cues to guide generation. Experiments on the DreamBooth and Custom101 benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches, achieving a better balance between semantic fidelity and subject consistency.

Problem

• subject-driven image customization
• text-to-image generation
• identity preservation
• reference subject
• attention distribution

Innovation

• Attention Distribution Shift
• Subject-Driven Image Customization
• Conditional Attention
• Dual-Branch Architecture
• Stable Diffusion