Masked Auto-Regressive Variational Acceleration: Fast Inference Makes Practical Reinforcement Learning

2025-11-19

AI Summary

To address the slow inference and RL incompatibility of Masked Autoregressive Diffusion (MAR) models—caused by their hierarchical inference (outer autoregressive mask unmasking + inner multi-step diffusion)—this paper proposes MARVAL. MARVAL employs variational acceleration to compress the diffusion denoising chain into a single-step generation while preserving MAR’s flexible, mask-guided decoding order. Its core innovation is a score-function-based variational distillation objective, enabling the first joint optimization of single-step efficient generation and RL fine-tuning for MAR models. The method integrates masked autoregressive modeling, diffusion distillation, score-function optimization, and reward alignment. On ImageNet 256×256, MARVAL achieves an FID of 2.00, accelerates inference by over 30×, and significantly outperforms baselines in CLIP Score and image reward—demonstrating superior trade-offs among generation quality, efficiency, and human preference alignment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Masked auto-regressive diffusion models (MAR) benefit from the expressive modeling ability of diffusion models and the flexibility of masked auto-regressive ordering. However, vanilla MAR suffers from slow inference due to its hierarchical inference mechanism: an outer AR unmasking loop and an inner diffusion denoising chain. Such decoupled structure not only harm the generation efficiency but also hinder the practical use of MAR for reinforcement learning (RL), an increasingly critical paradigm for generative model post-training.To address this fundamental issue, we introduce MARVAL (Masked Auto-regressive Variational Acceleration), a distillation-based framework that compresses the diffusion chain into a single AR generation step while preserving the flexible auto-regressive unmasking order. Such a distillation with MARVAL not only yields substantial inference acceleration but, crucially, makes RL post-training with verifiable rewards practical, resulting in scalable yet human-preferred fast generative models. Our contributions are twofold: (1) a novel score-based variational objective for distilling masked auto-regressive diffusion models into a single generation step without sacrificing sample quality; and (2) an efficient RL framework for masked auto-regressive models via MARVAL-RL. On ImageNet 256*256, MARVAL-Huge achieves an FID of 2.00 with more than 30 times speedup compared with MAR-diffusion, and MARVAL-RL yields consistent improvements in CLIP and image-reward scores on ImageNet datasets with entity names. In conclusion, MARVAL demonstrates the first practical path to distillation and RL of masked auto-regressive diffusion models, enabling fast sampling and better preference alignments.

Problem

• Slow inference in masked auto-regressive diffusion models due to hierarchical structure
• Practical limitations for reinforcement learning applications in generative models
• Need to accelerate generation while preserving sample quality and flexibility

Innovation

• Distills diffusion chain into single AR generation step
• Uses score-based variational objective for distillation
• Enables efficient RL framework for masked auto-regressive models