Shift-and-Sum Quantization for Visual Autoregressive Models

2026-06-14

AI Summary

This work addresses significant challenges in post-training quantization (PTQ) of Vision Autoregressive models (VARs), specifically the large reconstruction error in attention value products and the mismatch between codebook entry sampling frequencies and prediction probabilities. To overcome these issues, the paper introduces the first PTQ framework tailored explicitly for VARs, featuring a novel shifted-sum quantization scheme that substantially reduces attention reconstruction error under coarse quantization granularity. Furthermore, it integrates symmetric shifted token aggregation with a calibration data resampling strategy to effectively align the empirical codebook usage distribution with the model’s predictive distribution. The proposed method achieves state-of-the-art performance across multiple generative tasks—including class-conditional generation, image inpainting, outpainting, and editing—significantly surpassing existing PTQ approaches and setting a new benchmark for quantized VAR models.

Abstract

Post-training quantization (PTQ) enables efficient deployment of deep networks using a small set of data. Its application to visual autoregressive models (VAR), however, remains relatively unexplored. We identify two key challenges for applying PTQ to VAR: (i) large reconstruction errors in attention-value products, especially at coarse scales where high attention scores occur more frequently; and (ii) a discrepancy between the sampling frequencies of codebook entries and their predicted probabilities due to limited calibration data. To address these challenges, we propose a PTQ framework tailored for VAR. First, we introduce a shift-and-sum quantization method that reduces reconstruction errors by aggregating quantized results from symmetrically shifted duplicates of value tokens. Second, we present a resampling strategy for calibration data that aligns sampling frequencies of codebook entries with their predicted probabilities. Experiments on class-conditional image generation, inpainting, outpainting, and class-conditional editing show consistent improvements across VAR architectures, establishing a new state of the art in PTQ for VAR.

Problem

• post-training quantization
• visual autoregressive models
• reconstruction error
• attention-value products
• codebook sampling discrepancy

Innovation

• Shift-and-Sum Quantization
• Post-training Quantization
• Visual Autoregressive Models
• Calibration Resampling
• Attention-value Reconstruction