TempAct: Advancing Temporal Plausibility in Autoregressive Video Generation via Planner-Executor RL

2026-06-26

AI Summary

This work addresses the challenges in autoregressive video generation, where chunk-based strategies often lead to ambiguous temporal instructions, action delays, and error propagation—issues that are difficult to mitigate through supervised fine-tuning or distillation. To overcome these limitations, the authors propose TempAct, a novel framework that synergistically integrates a large language model planner with an autoregressive diffusion executor. TempAct introduces a planning-execution cooperative reinforcement learning architecture, leveraging hierarchical population-based exploration and a multi-granularity reward mechanism to enable precise credit assignment. Evaluated on the Self-Forcing and LongLive datasets, the method significantly improves temporal consistency and response latency while preserving high-fidelity visual generation quality.

Abstract

Autoregressive (AR) video diffusion models enable low-latency streaming generation by synthesizing videos chunk by chunk with cached visual context, but this chunk-wise formulation makes temporal instruction following ambiguous. A single global prompt does not specify which sub-event should be realized in each chunk, while naively switching to step-wise prompts often leads to delayed reactions, blended step semantics, and error propagation across prompt transitions. These failures are difficult to address with supervised fine-tuning or distillation alone: SFT suffers from exposure bias, while rollout-based distillation still optimizes low-level denoising or teacher-distribution matching rather than directly enforcing action ordering and prompt-transition correctness. We address these challenges with TempAct, a planner--executor reinforcement learning framework that jointly optimizes temporal decomposition and step-conditioned execution for temporally plausible AR video generation. TempAct uses an LLM planner to explore span-aware step prompts that are executable by the video model, and trains an AR diffusion executor to follow these prompts under its own generated histories. Its key mechanism is hierarchical group exploration: candidate plans form planning groups, and each plan induces an execution group of multiple continuations from a shared visual context, enabling plan-level credit assignment for long-horizon temporal outcomes and executor-level credit assignment for prompt-switch behavior. We further design hierarchical rewards that combine plan-quality and full-video temporal feedback for the planner with local transition-level step-following rewards, aesthetic regularization, and KL constraints for the executor. Experiments on Self-Forcing and LongLive show that TempAct improves temporal consistency while preserving overall visual quality.

Problem

• temporal plausibility
• autoregressive video generation
• temporal instruction following
• prompt transition
• chunk-wise generation

Innovation

• Temporal Plausibility
• Autoregressive Video Generation
• Planner-Executor RL
• Hierarchical Group Exploration
• Span-aware Prompting