Beyond Text Conditioning: A Systematic Study of MLLM-DiT Fusion for Video Generation

2026-08-14

AI Summary

This study addresses the insufficient high-level semantic planning in Diffusion Transformers for video generation by proposing BiVidGen, a novel framework that introduces an explicit semantic planning paradigm bridged by discrete visual tokens generated via Multimodal Large Language Models. By integrating an EMA tokenizer, autoregressive modeling, and multi-layer cross-attention mechanisms, the method achieves dual-conditioned rendering with both textual and visual tokens. As a systematic exploration of MLLM-DiT integration, BiVidGen significantly improves semantic alignment and temporal coherence on VBench-Long, outperforming fine-tuned DiT baselines and effectively enhancing semantic controllability in long-form video synthesis.

Abstract

Diffusion Transformers (DiTs) have become the dominant paradigm for high-fidelity video generation, yet their ability to perform high-level semantic planning remains limited. While hybrid architectures integrating MLLMs with diffusion backbones have shown strong advantages in image synthesis, such designs remain underexplored in video generation, where existing approaches often treat MLLMs primarily as frozen feature encoders rather than semantic generators. To fill this gap, we systematically study how an MLLM should be integrated with a DiT for video generation by answering three questions: what intermediate representation should bridge the MLLM and DiT, how the MLLM should generate it, and how the DiT should incorporate it during diffusion rendering. Our analysis reveals three key findings: (1) discrete semantic visual tokens produced by an EMA-based tokenizer provide a stable and expressive interface, (2) autoregressive causal modeling is effective for generating these tokens, and (3) explicit visual-token conditioning is more effective than prompt refinement or latent bridging. Based on these findings, we propose BiVidGen, a hybrid framework where an MLLM first generates semantic visual tokens and a DiT renders videos conditioned on both text and these tokens via multi-layer cross-attention. Extensive experiments show that BiVidGen improves semantic alignment and temporal coherence over a fine-tuned DiT baseline, achieving stronger performance on VBench-Long. These results demonstrate that explicit MLLM-based visual planning provides an effective intermediate interface for text-to-video generation beyond text-only conditioning.

Problem

• Video Generation
• Diffusion Transformers
• Multimodal Large Language Models
• Semantic Planning
• MLLM-DiT Fusion

Innovation

• MLLM-DiT Fusion
• Semantic Visual Tokens
• Autoregressive Causal Modeling
• Explicit Visual-Token Conditioning
• BiVidGen