generative models

This work addresses the challenge of high-quality multimodal video generation and editing by proposing a unified multimodal foundation model architecture. Methodologically, it introduces variable-aspect-ratio 1080p video latent-space modeling, cross-modal alignment training across text, image, video, and audio modalities, efficient tokenization, large-scale parallel training and inference optimization, and a rigorously quality-controlled data curation strategy coupled with a novel evaluation protocol. Key contributions include the first 30-billion-parameter video generation model supporting long-horizon generation (73K tokens, i.e., 16 seconds at 16 fps), instruction-driven precise editing, user-provided image personalization, and synchronized audio-video synthesis. The model achieves state-of-the-art performance across five benchmarks: text-to-video, video personalization, video editing, video-to-audio, and text-to-audio—demonstrating substantial improvements in temporal coherence and semantic controllability.

This work addresses the inefficiency and poor visual fidelity of autoregressive video generation models by proposing NOVA—the first end-to-end autoregressive video generation framework (0.6B parameters) that eliminates vector quantization. Methodologically, NOVA jointly models inter-frame temporal causal prediction and intra-frame spatial set prediction: it employs GPT-style unidirectional temporal attention to ensure strict causality and introduces bidirectional intra-frame attention to enhance spatial coherence. Crucially, it abandons conventional VQ-based tokenization, enabling direct autoregressive modeling in the native pixel space. Experiments demonstrate that NOVA surpasses existing autoregressive methods across data efficiency, inference speed, motion smoothness, and visual quality. Moreover, its text-to-image generation performance matches leading diffusion models, while requiring significantly lower training costs. NOVA also supports long-video synthesis and unifies zero-shot multi-task capabilities within a single architecture.

Current video diffusion models are computationally constrained, limiting generation to ~10-second clips; autoregressive extension methods suffer from abrupt scene transitions, unnatural motion, and error accumulation due to naive frame concatenation. To address this, we propose a progressive frame-level noise scheduling scheme coupled with a fine-grained denoising-and-shift strategy, relaxing the conventional single-noise-level assumption. Within a sliding attention window, our approach enables smooth inter-frame attention alignment and persistent cross-segment information propagation. To our knowledge, this is the first method enabling text-to-video synthesis of 60-second (1,440-frame) sequences. It achieves significantly improved temporal consistency and visual fidelity, with minimal quality degradation along the time axis—matching the performance of state-of-the-art short-video diffusion models.

Existing video generation methods—particularly diffusion-based approaches—struggle to model long-term narrative structure and cross-shot character consistency, hindering cinematic-quality long-video synthesis. To address this, we propose a hierarchical long-video generation framework: an autoregressive visual token predictor operates at the top level to model global plot progression, while a conditional diffusion model at the bottom level ensures high-fidelity frame rendering. We introduce, for the first time, multimodal script encoding to explicitly enforce cross-scene consistency in character appearance, motion, and stylistic attributes. This architecture decouples narrative reasoning from visual synthesis, enabling minute-scale video generation. Evaluated on diverse cinematic datasets, our method significantly improves generated video length, narrative coherence, and visual fidelity, achieving state-of-the-art performance.

Current text-to-video models suffer from significant deficiencies in physical plausibility, photorealistic lighting, camera motion, and temporal coherence, limiting their applicability to cinematic-grade synthesis. To address this, we propose the first multi-agent VLM framework tailored for high-fidelity 3D video generation, featuring decoupled Director, Programmer, and Reviewer agents. Our method decomposes the synthesis task, automatically generates Blender scripting code, and performs iterative optimization guided by vision-language feedback—enabling end-to-end, interpretable, and editable video generation. Deeply integrating cinematographic knowledge with a closed-loop 3D rendering pipeline, it produces high-fidelity videos fully aligned with textual prompts—without manual intervention. Experiments demonstrate superior performance over leading commercial models across five video quality and instruction-following metrics. User studies further confirm substantial improvements: +28.6% in physical plausibility, +31.2% in temporal consistency, and higher overall quality scores.

Existing approaches to automatic video trailer generation predominantly rely on rule-based clip selection, which struggles to produce semantically coherent and emotionally compelling narratives. This work systematically reviews the technological evolution from heuristic selection to generative synthesis and introduces a novel taxonomy for AI-driven trailer generation tailored to the era of foundation models. We propose a unified framework that integrates autoregressive Transformers, multimodal large language models (MLLMs), text-to-video diffusion models (e.g., Sora, Veo), and graph convolutional networks (GCNs). This architecture enables controllable generation and semantic restructuring, substantially enhancing content creation efficiency on user-generated content (UGC) platforms, while also highlighting the ethical challenges posed by high-fidelity neural synthesis.

Existing generative models struggle to meet the stringent requirements of professional educational videos—namely, domain-specific accuracy, visually structured presentation, and logical coherence. To address this, we propose Code2Video, the first educational video generation framework centered on executable Python code, which explicitly encodes content structure, visual layout, and transition logic. Our method introduces a novel tri-agent collaboration mechanism (planning–coding–review), a vision-language model (VLM) evaluator enhanced with visual anchors, and TeachQuiz—a quantitative metric designed specifically for pedagogical effectiveness. Integrating instruction-driven code generation, scope-aware automatic code repair, and end-to-end code-to-video rendering, Code2Video significantly improves controllability and interpretability. Evaluated on the MMMC benchmark, it outperforms baseline methods by 40% and produces video quality approaching that of human-crafted tutorials.

This work addresses the limited controllability of temporal dynamics and editing in existing video diffusion Transformer models. To overcome this, the authors propose a lightweight temporal control module that enables explicit and fine-grained manipulation of motion speed and temporal structure without altering the pre-trained DiT backbone. By effectively leveraging the generative priors learned during pre-training, the method significantly enhances temporal controllability in video generation while preserving the original output quality. The approach thus offers a practical and efficient solution for precise temporal editing in diffusion-based video synthesis.

This work addresses the high computational cost of existing diffusion models in video generation, which hinders real-time interactive applications. The authors propose an efficient method for synthesizing videos of static scenes by first generating sparse keyframes with a diffusion model, then leveraging 3D reconstruction and differentiable rendering to interpolate a full video sequence. A novel camera trajectory-aware adaptive keyframe scheduling mechanism dynamically adjusts keyframe density to preserve geometric consistency throughout the sequence. The approach achieves high visual fidelity and temporal stability while accelerating 20-second video generation by over 40× compared to baseline diffusion models.

Existing video generation methods predominantly operate in the 2D pixel space without explicit 3D structural constraints, leading to geometric temporal inconsistency, physically implausible motion, and structural artifacts. To address this, we propose a latent-space geometry-aware video generation framework built upon latent diffusion models. Our method integrates a frame-wise depth prediction module and introduces a multi-view geometric loss that aligns predicted depth maps across frames within a shared 3D coordinate system—enabling joint optimization of appearance synthesis and 3D structural modeling. Leveraging a diffusion Transformer architecture, we unify a depth prediction network with an image-level latent encoder and impose latent-space depth regularization. Extensive experiments demonstrate that our approach significantly improves geometric consistency, temporal stability, and physical plausibility of generated videos across multiple benchmarks, outperforming current state-of-the-art methods.