Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding

2026-08-09

AI Summary

This work addresses the inefficiency of intermediate feature transmission in distributed deployments of large vision foundation models, where bandwidth and computational constraints are exacerbated by existing compression methods that neglect the two-dimensional spatial structure of Patch Tokens. To overcome this limitation, the authors propose the Visual Token Codec (VTC), which explicitly models spatial redundancy among Patch Tokens for the first time. VTC employs a dual-path architecture to separately process global and local tokens and integrates a spatial-channel contextual entropy model, factorized priors, and a feature-matching supervision mechanism. This design enables intermediate-layer compression and variable-rate adaptation within a single model. Evaluated on DINOv2 and SAM, VTC achieves 90% of the original performance across classification, segmentation, and detection tasks while reducing bitrate by 15.7–37.4×, substantially outperforming current approaches.

Abstract

Distributed deployment of large vision foundation models often partitions a ViT backbone and exchanges intermediate token features between computing nodes, making efficient feature compression critical under bandwidth and computation constraints. Existing ViT feature codecs typically flatten heterogeneous global and patch tokens into an L x C pseudo image, causing entropy models to mainly capture sequence-axis dependencies while overlooking the native two-dimensional patch-grid structure. In this paper, we show that ViT patch tokens retain strong local spatial correlations on the original grid. To exploit this structural prior, we propose the Visual Token Codec (VTC), a dual-path learned codec that separates global and patch tokens into dedicated coding paths. Global tokens are compressed with a lightweight factorized prior, whereas patch tokens are encoded on the patch-token grid using a spatial-channel context entropy model. To support intermediate-layer compression and practical rate adaptation, VTC further incorporates feature-matching supervision after subsequent ViT blocks and variable-rate modules within a single codec. Experiments on DINOv2 and SAM3 show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks. At 90% of uncompressed-feature performance, VTC reduces bitrate by 15.7x-37.4x across these tasks. We further provide intermediate-layer rate-utility analyses for practical transmission- and storage-oriented deployment scenarios.

Problem

• ViT feature compression
• distributed deployment
• spatial redundancy
• intermediate token features
• bandwidth constraints

Innovation

• Visual Token Codec
• spatial redundancy
• ViT feature compression
• dual-path architecture
• spatial-channel context