VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling

2026-08-09

AI Summary

This work addresses the computational and memory bottlenecks in vision-language models caused by the quadratic complexity of self-attention when processing extremely long interleaved image-text sequences. To overcome this limitation, the authors propose VLZip, a framework that achieves joint visual-textual compression within a pure Transformer architecture for the first time. VLZip employs hierarchical distillation to generate layer-specific “soft prefixes,” which are injected into the hidden states of each decoder layer, substantially reducing attention sequence length while preserving fine-grained global context. The study introduces the layer-specific soft prefix mechanism, establishes LongVLBench—the first long-context multimodal benchmark tailored for narrative-level reasoning—and demonstrates scalability up to 120K tokens during training and over 280K tokens (extendable to 2M) at inference, achieving state-of-the-art performance under extreme long-context conditions with significantly reduced memory overhead.

Abstract

Vision Language Models (VLMs) face significant challenges with ultra-long, interleaved image-text sequences due to the quadratic complexity of self-attention. Current solutions either resort to aggressive token pruning, risking irreversible information loss, or adopt efficient but less precise architectures, while largely ignoring the equally vital textual component. We introduce VLZip, a framework that unifies visual and textual compression for high-fidelity reasoning within a pure Transformer. At its core, VLZip hierarchically distills visual and textual segments into compact, layer-specific "soft prefixes" and injects them into each decoder layer's hidden states, drastically shortening the attention sequence while preserving fine-grained global context. To address deficient evaluations in the field, we also introduce LongVLBench, a new benchmark derived from video narratives that demands holistic, narrative-level reasoning. Extensive experiments show VLZip achieves leading performance on long-context multimodal reasoning, enabling training up to 120K tokens, a 6x increase over the baseline, and inference beyond 280K tokens with significantly reduced memory, while demonstrating the memory scalability to handle up to 2M tokens. By excelling at extreme context lengths where existing methods collapse, VLZip establishes an efficient and powerful new standard for long-context multimodal AI. Code is available at https://github.com/ShareLab-SII/VLZip.

Problem

• Vision Language Models
• long-context modeling
• interleaved image-text sequences
• self-attention complexity
• multimodal compression

Innovation

• visual-textual compression
• soft prefixes
• long-context multimodal modeling
• hierarchical distillation
• memory scalability