UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation

2026-05-12

AI Summary

This work addresses the challenge of subject identity confusion and attribute leakage in multi-reference image generation, which arises from existing methods’ decoupled treatment of semantic and appearance features. To overcome this limitation, the authors propose UniCustom, a unified visual conditioning framework that uniquely integrates ViT-extracted semantic features with VAE-encoded appearance features before feeding them into a vision-language model (VLM). The integration is achieved through a lightweight linear fusion layer, complemented by a two-stage training strategy—comprising reconstruction pretraining followed by supervised fine-tuning—and a slot-binding regularization mechanism to mitigate cross-reference ambiguity. Evaluated on two established multi-reference generation benchmarks, UniCustom demonstrates significant improvements in subject consistency, instruction following, and compositional fidelity, consistently outperforming strong existing baselines.

Abstract

Multi-reference image generation aims to synthesize images from textual instructions while faithfully preserving subject identities from multiple reference images. Existing VLM-enhanced diffusion models commonly rely on decoupled visual conditioning: semantic ViT features are processed by the VLM for instruction understanding, whereas appearance-rich VAE features are injected later into the diffusion backbone. Despite its intuitive design, this separation makes it difficult for the model to associate each semantically grounded subject with visual details from the correct reference image. As a result, the model may recognize which subject is being referred to, but fail to preserve its identity and fine-grained appearance, leading to attribute leakage and cross-reference confusion in complex multi-reference settings. To address this issue, we propose UniCustom, a unified visual conditioning framework that fuses ViT and VAE features before VLM encoding. This early fusion exposes the VLM to both semantic cues and appearance-rich details, enabling its hidden states to jointly encode the referred subject and corresponding visual appearance with only a lightweight linear fusion layer. To learn such unified representations, we adopt a two-stage training strategy: reconstruction-oriented pretraining that preserves reference-specific appearance details in the fused hidden states, followed by supervised finetuning on single- and multi-reference generation tasks. We further introduce a slot-wise binding regularization that encourages each image slot to preserve low-level details of its corresponding reference, thereby reducing cross-reference entanglement. Experiments on two multi-reference generation benchmarks demonstrate that UniCustom consistently improves subject consistency, instruction following, and compositional fidelity over strong baselines.

Problem

• multi-reference image generation
• visual conditioning
• subject identity preservation
• attribute leakage
• cross-reference confusion

Innovation

• unified visual conditioning
• early feature fusion
• multi-reference image generation
• slot-wise binding regularization
• two-stage training