Bernini: Latent Semantic Planning for Video Diffusion

2026-05-21

AI Summary

This work addresses the challenge of achieving controllable, high-fidelity video editing by effectively integrating the semantic reasoning of multimodal large language models (MLLMs) with the high-quality generative capabilities of diffusion models. The authors propose Bernini, a novel framework that decouples semantic planning from pixel-level rendering for the first time: an MLLM generates a semantic plan via chain-of-thought reasoning in the Vision Transformer (ViT) embedding space, which is then fused by a DiT-based renderer with text prompts and source VAE features to synthesize the output video. To align multiple visual inputs, the method introduces segment-aware 3D rotary positional encoding, enabling modular training and lightweight coordination between components. Experiments demonstrate that Bernini achieves state-of-the-art performance across multiple video generation and editing benchmarks, exhibiting exceptional generalization and fine-grained detail preservation, particularly in complex editing scenarios.

Abstract

Multimodal large language models (MLLMs) and diffusion models have each reached remarkable maturity: MLLMs excel at reasoning over heterogeneous multimodal inputs with strong semantic grounding, while diffusion models synthesize images and videos with photorealistic fidelity. We argue that these two families can be unified through a simple division of labor: MLLMs perform semantic planning, while diffusion models render pixels from high-level semantic guidance and low-level visual features. Building on this idea, we propose Bernini, a unified framework for video generation and editing. An MLLM-based planner predicts the target semantic representation directly in the ViT embedding space, and a DiT-based renderer synthesizes pixels conditioned on this plan, augmented by text features and, for editing, source VAE features for detail preservation. Because semantics serve as the interface, the planner and renderer can be trained separately and only lightly co-trained, preserving the pretrained strengths of both components while keeping training efficient. To better handle multiple visual inputs, we introduce Segment-Aware 3D Rotary Positional Embedding (SA-3D RoPE), and further incorporate chain-of-thought reasoning in the planner to better transfer understanding into generation. Bernini achieves state-of-the-art performance across a wide range of video generation and editing benchmarks, with the MLLM's pretrained understanding translating into strong generalization on challenging editing tasks.

Problem

• video generation
• video editing
• multimodal large language models
• diffusion models
• semantic planning

Innovation

• semantic planning
• video diffusion
• multimodal LLM
• DiT-based rendering
• SA-3D RoPE