MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts

2026-06-30

AI Summary

This work addresses the optimization conflicts and semantic error propagation inherent in visual autoregressive (VAR) models for multiscale representation learning, where shared architectures struggle to simultaneously capture global semantics at coarse scales and fine-grained details at high resolutions. To resolve this, the authors propose a scale-aware Mixture-of-Experts (MoE) architecture with token routing that enables scale-adaptive expert selection, thereby decoupling representations across scales. Additionally, they introduce a residual feature alignment strategy tailored to the VAR paradigm, which integrates external self-supervised features to strengthen early-stage semantic modeling. Evaluated on ImageNet at 256×256 resolution, the method significantly outperforms dense baselines in terms of FID, achieving superior performance with fewer parameters and training epochs, and the performance gap widens as training progresses.

Abstract

Visual AutoRegressive modeling (VAR) has pioneered a coarse-to-fine multi-scale autoregressive generative paradigm, demonstrating strong capabilities in image generation. However, VAR still suffers from inherent deficiencies in multi-scale representation learning. Specifically, lower scales primarily capture global semantics, while higher scales focus on fine-grained details. Employing a shared architecture across scales induces optimization conflicts. Moreover, due to the causal autoregressive process, inaccurate semantics at early scales can propagate and significantly degrade the final output. To address these issues, we introduce a scale-aware token-routed Mixture of Experts (MoE) architecture, allowing scale-adaptive expert selection, thereby facilitating decoupled representation learning across scales. In addition, we enhance semantic modeling at early scales by incorporating external self-supervised features. Unlike naive alignment, we analyse and design a residual feature aggregation scheme tailored to the VAR paradigm. Extensive experiments show that our method significantly improves both training efficiency and generation quality. On the ImageNet 256*256 benchmark, our model achieves a superior FID compared to the dense baseline while requiring only half of the default training epochs and a smaller parameter budget, with a merely marginal increase in training cost. Moreover, the performance gap further widens with larger training epochs.

Problem

• Visual Autoregressive Modeling
• Multi-Scale Representation
• Mixture of Experts
• Semantic Propagation
• Representation Learning

Innovation

• Mixture of Experts
• Multi-Scale Representation
• Visual Autoregressive Modeling
• Scale-aware Routing
• Residual Feature Aggregation