PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

2025-06-19

AI Summary

Visual generative models suffer from quadratic computational and memory overheads in attention mechanisms, especially for high-resolution image and video generation. Existing sparse and low-bit quantization techniques degrade sharply under low density and narrow bit-widths. To address this, we propose Pattern-Aware Token Reordering (PARO), which analyzes diverse visual attention patterns and restructures them into hardware-friendly block-wise layouts—rather than adapting to existing sparse or quantized attention schemes. Our method integrates PARO with structured sparse attention, INT8/INT4 quantization, and block-aware scheduling. Experiments demonstrate that our approach preserves full-precision baseline quality (lossless metrics) across image and video generation tasks, achieves 20–30% attention density, and delivers 1.9×–2.7× end-to-end latency reduction. This significantly improves deployment efficiency and hardware compatibility.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Hardware-aware ML
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

In visual generation, the quadratic complexity of attention mechanisms results in high memory and computational costs, especially for longer token sequences required in high-resolution image or multi-frame video generation. To address this, prior research has explored techniques such as sparsification and quantization. However, these techniques face significant challenges under low density and reduced bitwidths. Through systematic analysis, we identify that the core difficulty stems from the dispersed and irregular characteristics of visual attention patterns. Therefore, instead of introducing specialized sparsification and quantization design to accommodate such patterns, we propose an alternative strategy: *reorganizing* the attention pattern to alleviate the challenges. Inspired by the local aggregation nature of visual feature extraction, we design a novel **Pattern-Aware token ReOrdering (PARO)** technique, which unifies the diverse attention patterns into a hardware-friendly block-wise pattern. This unification substantially simplifies and enhances both sparsification and quantization. We evaluate the performance-efficiency trade-offs of various design choices and finalize a methodology tailored for the unified pattern. Our approach, **PAROAttention**, achieves video and image generation with lossless metrics, and nearly identical results from full-precision (FP) baselines, while operating at notably lower density (~20%-30%) and bitwidth (**INT8/INT4**), achieving a **1.9x** to **2.7x** end-to-end latency speedup.

Problem

• Reduces memory and computational costs in visual generation models
• Addresses challenges of sparse and quantized attention patterns
• Improves efficiency without sacrificing generation quality

Innovation

• Pattern-Aware token ReOrdering (PARO) technique
• Unifies attention into block-wise pattern
• Enables efficient INT8/INT4 quantization