Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment

2026-07-05

AI Summary

This work addresses the challenges of maintaining identity consistency and modeling complex interactions among multiple subjects in controllable video generation by proposing the Aura framework. Aura introduces AI-director-level captioning to describe scene dynamics and leverages a vision-language model to extract multimodal semantic features, which are then mapped into a diffusion Transformer (DiT) latent space via a two-stage alignment strategy. The method innovatively incorporates a subject-aware RoPE-Shift mechanism, learnable subject tokens, memory tokens, and a Progressive-APG inference strategy to enable high-fidelity, semantically coherent multi-subject video synthesis. Experimental results demonstrate that Aura significantly outperforms existing approaches on both single- and multi-subject generation tasks, effectively suppressing copy-paste artifacts while enhancing identity consistency and overall visual quality.

Abstract

Subject-driven and multi-element video generation are central to controllable video synthesis, but existing methods still struggle to preserve identity consistency and model complex relationships among multiple subjects. In this paper, we propose Aura, a unified framework for high-fidelity and identity-consistent video generation. To better capture scene dynamics and subject interactions, we introduce AI director-level captions that provide dense and structured descriptions of video content. We further leverage a vision-language model (VLM) with learnable queries to extract multimodal semantic features from textual and visual references, covering both global semantics and fine-grained visual cues. To bridge the representational gap between the VLM and the Diffusion Transformer (DiT), we design a two-stage alignment strategy that progressively maps VLM features into the DiT feature space. For visual conditioning, we adopt token concatenation to inject reference information directly into the generation process. To distinguish heterogeneous subject types and reduce common copy-paste artifacts, we develop a subject-aware RoPE-Shift mechanism. To further differentiate reference images of different categories, we introduce subject-aware learnable tokens. In addition, we introduce Memory Tokens to balance the training signal across examples with different numbers of reference subjects. During inference, Progressive-APG (Adaptive Prompt Guidance) further alleviates oversaturation and improves semantic alignment with user prompts. Finally, we build a high-quality video-subject image dataset through a dedicated data construction pipeline. Extensive experiments show that our method achieves state-of-the-art performance on both single-subject generation and more challenging multi-element scenarios.

Problem

• identity consistency
• multi-subject video generation
• subject-driven generation
• complex subject relationships
• controllable video synthesis

Innovation

• Vision-Language Model
• Diffusion Transformer
• Semantic Alignment
• Subject-Aware Generation
• Multi-Subject Video Synthesis