Markovian Scale Prediction: A New Era of Visual Autoregressive Generation

2025-11-28

AI Summary

Visual autoregressive models (VARs) suffer from prohibitive computational cost and excessive GPU memory consumption due to full-context dependency, severely limiting scalability. To address this, we propose Markov-VAR—the first VAR framework incorporating the Markov assumption into visual autoregressive generation. It constructs a compact, dynamic state representation via a sliding window and history vector compression, replacing global context modeling with localized state evolution. This design achieves substantial efficiency gains without compromising generation quality: on ImageNet 256×256, it improves FID by 10.5%; at 1024×1024 resolution, peak GPU memory is reduced by 83.8%. Our core contribution lies in pioneering the application of Markov process modeling to visual autoregression, enabling joint optimization of generation fidelity and computational efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Visual AutoRegressive modeling (VAR) based on next-scale prediction has revitalized autoregressive visual generation. Although its full-context dependency, i.e., modeling all previous scales for next-scale prediction, facilitates more stable and comprehensive representation learning by leveraging complete information flow, the resulting computational inefficiency and substantial overhead severely hinder VAR's practicality and scalability. This motivates us to develop a new VAR model with better performance and efficiency without full-context dependency. To address this, we reformulate VAR as a non-full-context Markov process, proposing Markov-VAR. It is achieved via Markovian Scale Prediction: we treat each scale as a Markov state and introduce a sliding window that compresses certain previous scales into a compact history vector to compensate for historical information loss owing to non-full-context dependency. Integrating the history vector with the Markov state yields a representative dynamic state that evolves under a Markov process. Extensive experiments demonstrate that Markov-VAR is extremely simple yet highly effective: Compared to VAR on ImageNet, Markov-VAR reduces FID by 10.5% (256 $ imes$ 256) and decreases peak memory consumption by 83.8% (1024 $ imes$ 1024). We believe that Markov-VAR can serve as a foundation for future research on visual autoregressive generation and other downstream tasks.

Problem

• VAR models suffer from computational inefficiency due to full-context dependency
• Markov-VAR eliminates full-context dependency while improving performance
• The method compresses previous scales into compact history vectors

Innovation

• Markov-VAR introduces non-full-context Markov process
• Sliding window compresses previous scales into history vector
• Dynamic state evolution reduces memory and improves performance