VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

2026-07-14

AI Summary

This work addresses the high latency and substantial memory overhead of current vision-language models caused by processing a large number of visual tokens, as well as the performance degradation or disruptive retraining often introduced by existing compression methods. The authors propose VisCo, a training-efficient self-compression framework that uniquely leverages the pre-trained vision-language model itself as an intrinsic encoder. By employing a parameter-shared autoencoding architecture and a minimal set of memory tokens, VisCo compresses visual information while preserving hierarchical features across encoding and decoding stages. Requiring neither additional modules nor extensive retraining, VisCo consistently outperforms existing approaches across all compression ratios, remains stable even in extreme single-token settings, and further enhances the original model’s performance through its compressed memory tokens.

Abstract

Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference latency and memory overhead. This has motivated extensive research on visual token compression. While training-free strategies rely on heuristic metrics and suffer significant performance degradation under high compression ratios, many training-based methods introduce external compression modules that force the VLM backbone to adapt, incurring substantial retraining cost and compromising VLMs' priors. Effective visual token compression hinges on strong information encoding, a capability already present in pretrained VLMs but underutilized by existing approaches. Motivated by this, we propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor. VisCo is a parameter-sharing autoencoder that compresses visual information using a small set of memory tokens and transfers hierarchical information from encoding to decoding. Experiments show that VisCo surpasses prior methods across all evaluated compression ratios, with larger gains under more aggressive compression, and remains stable even in the extreme single-token setting. Moreover, when combined with the original visual tokens, the learned memory tokens can even improve the base model, suggesting that VisCo captures complementary representations beyond compression.

Problem

• visual token compression
• vision-language models
• inference latency
• memory overhead
• compression ratio

Innovation

• visual token compression
• vision-language models
• intrinsic encoder
• memory tokens
• self-compression