Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption

2026-07-01

AI Summary

This work addresses the challenge of excessive memory consumption and reduced inference throughput in autoregressive video generation, where Key-Value (KV) cache grows linearly with sequence length. Existing compression techniques often compromise long-range temporal dependencies, leading to visual artifacts such as flickering and identity inconsistency. To overcome this, the authors propose Instance-Specific Parameter Absorption (ISPA), a framework that reframes KV cache compression as a parameter modulation problem. ISPA selectively switches certain layers to local attention and distills historical context into model weights via a closed-form least-squares solution, thereby achieving substantial cache reduction without sacrificing generation fidelity. Evaluated across models ranging from 1.3B to 14B parameters, ISPA reduces KV cache usage by up to 50% while preserving near-lossless visual consistency and generation quality.

Abstract

Autoregressive (AR) streaming models have emerged as a powerful paradigm for long video generation. However, the linearly growing Key-Value (KV) cache poses a significant bottleneck, leading to memory overload and degraded inference throughput. A common compression method is to drop redundant KV tokens, which often breaks long-range dependencies, resulting in temporal flickering and identity loss. In this paper, we propose Instance-Specific Parametric Absorption (ISPA), a novel framework that shifts the KV cache compression from discarding to distilling. The core idea is to transit a subset of layers from Full-Attention (F-Layers) to memory-efficient Local-Attention (L-Layers) by "absorbing" historical context into the model's weights. Specifically, during a brief warmup phase, ISPA monitors the output discrepancy between global and local attention. At the transition point, we solve a closed-form least-squares problem to compute an instance-specific weight modulation that compensates for the missing history. Experiments across architectures (1.3B to 14B) demonstrate that ISPA can remove up to 50\% of the KV cache with near-lossless visual quality. We hope this perspective encourages future work to explore parametric memory consolidation beyond external token-level cache management for streaming generative models.

Problem

• Autoregressive video generation
• KV cache
• Memory efficiency
• Long-range dependencies
• Temporal flickering

Innovation

• Instance-Specific Parametric Absorption
• KV Cache Compression
• Autoregressive Video Generation
• Memory-Efficient Attention
• Parametric Memory Consolidation