XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation

2025-06-26

AI Summary

This work addresses the challenge of jointly achieving fine-grained control over identity and semantic attributes (e.g., pose, style, illumination) in multi-subject text-to-image generation. We propose a reference-image-guided, token-level text-flow modulation method for DiT-based architectures. Specifically, a lightweight image-to-offset mapping network generates reference-driven modulation offsets for each text token, enabling disentangled modeling and independent controllability of identity and semantic attributes. Compared to existing approaches, our method significantly alleviates attribute entanglement and editing artifacts, thereby improving generation fidelity, cross-subject consistency, and editability. Extensive experiments demonstrate superior personalized control and synthesis quality—particularly in complex multi-subject scenarios—while maintaining computational efficiency and architectural compatibility with diffusion transformer backbones.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Achieving fine-grained control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, often undermines the editability and coherence of Diffusion Transformers (DiTs). Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose a novel multi-subject controlled generation model XVerse. By transforming reference images into offsets for token-specific text-stream modulation, XVerse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This advancement significantly improves personalized and complex scene generation capabilities.

Problem

• Control subject identity and attributes in multi-subject image generation
• Overcome artifacts and attribute entanglement in Diffusion Transformers
• Achieve high-fidelity editable multi-subject synthesis

Innovation

• DiT modulation for multi-subject control
• Token-specific text-stream offsets
• High-fidelity editable image synthesis