DiT as Real-Time Rerenderer: Streaming Video Stylization with Autoregressive Diffusion Transformer

2026-04-15

AI Summary

Existing diffusion-based video stylization methods struggle to balance spatiotemporal consistency and computational efficiency in long videos, and their multi-step denoising processes hinder real-time applications. This work proposes RTR-DiT, a streaming video stylization framework built upon the Diffusion Transformer that supports both text and reference image guidance. By introducing a reference-preserving key-value cache update strategy, the method enables stable long-video stylization with real-time style switching. Furthermore, through Self Forcing and distribution-matching distillation, a bidirectional teacher model is compressed into a low-latency autoregressive student model. The proposed approach outperforms existing methods in both text- and reference-guided tasks, achieving state-of-the-art results in quantitative metrics and visual quality, and for the first time enables high-quality, interactive real-time stylization of long videos.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Time-Series/Data Streams
• Natural Language Processing: Sentiment Analysis, Stylistic Analysis, and Argument Mining

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Recent advances in video generation models has significantly accelerated video generation and related downstream tasks. Among these, video stylization holds important research value in areas such as immersive applications and artistic creation, attracting widespread attention. However, existing diffusion-based video stylization methods struggle to maintain stability and consistency when processing long videos, and their high computational cost and multi-step denoising make them difficult to apply in practical scenarios. In this work, we propose RTR-DiT (DiT as Real-Time Rerenderer), a steaming video stylization framework built upon Diffusion Transformer. We first fine-tune a bidirectional teacher model on a curated video stylization dataset, supporting both text-guided and reference-guided video stylization tasks, and subsequently distill it into a few-step autoregressive model via post-training with Self Forcing and Distribution Matching Distillation. Furthermore, we propose a reference-preserving KV cache update strategy that not only enables stable and consistent processing of long videos, but also supports real-time switching between text prompts and reference images. Experimental results show that RTR-DiT outperforms existing methods in both text-guided and reference-guided video stylization tasks, in terms of quantitative metrics and visual quality, and demonstrates excellent performance in real-time long video stylization and interactive style-switching applications.

Problem

• video stylization
• diffusion models
• real-time processing
• long video consistency
• computational efficiency

Innovation

• Diffusion Transformer
• Video Stylization
• Autoregressive Model
• Knowledge Distillation
• Real-Time Rendering