Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation

2026-05-17

AI Summary

This work addresses the challenge of simultaneously achieving audio-visual identity consistency and synchronization in multi-speaker interaction scenarios by proposing Omni-Customizer, an end-to-end framework. The method introduces a novel Omni-Context Fusion module and a Masked TTS Cross-Attention mechanism, integrated with semantic-anchor-based multimodal RoPE positional encoding and a progressive cross-modal curriculum learning strategy to enable structured multimodal fusion while effectively mitigating voice leakage. Experimental results demonstrate that the proposed approach significantly outperforms existing methods in terms of visual identity similarity, voice timbre consistency, audio-visual synchronization, and overall generation fidelity.

Abstract

The landscape of joint audio and video generation has been fundamentally transformed by the advent of powerful foundation models. Despite these strides, achieving cohesive multimodal customization for the simultaneous preservation of visual identities and vocal timbres across multiple interacting subjects remains largely underexplored. To bridge this gap, we present Omni-Customizer, an end-to-end framework targeted at the precise binding and seamless fusion of multimodal identity information. Specifically, we introduce an Omni-Context Fusion (OCF) module that effectively enriches the base textual prompt with dense, multimodal identity cues, along with a Masked TTS Cross-Attention (MTP-CA) mechanism explicitly designed to prevent the severe "speech leakage" problem. Within this architecture, we propose Semantic-Anchored Multimodal RoPE (SA-MRoPE) to anchor visual and audio reference tokens, along with TTS embeddings, to their corresponding semantic descriptions, enabling structured multimodal fusion and robust identity binding. Furthermore, we devise a comprehensive training strategy that incorporates interleaved audio-video scheduling to rapidly adapt the audio branch to multilingual scenarios without degrading foundational priors, and a progressive in-pair to cross-pair curriculum to facilitate the learning of high-level and robust identity features. Extensive experiments demonstrate that Omni-Customizer achieves state-of-the-art performance in dual-modal customized generation, excelling across visual identity similarity, timbre consistency, precise audio-video synchronization, and overall video-audio fidelity.

Problem

• multimodal customization
• audio-video generation
• visual identity
• vocal timbre
• speech leakage

Innovation

• Omni-Customizer
• Multimodal Identity Binding
• Masked TTS Cross-Attention
• Semantic-Anchored RoPE
• End-to-End Audio-Video Generation