Post-Hoc Sparse Coding of Latent Communication Between Vision-Language Model Agents

2026-08-10

AI Summary

This work addresses the significant information redundancy and inefficient transmission inherent in current vision-language models that rely on dense latent-space tensor communication. The study demonstrates for the first time that latent representations exhibit strong posterior compressibility and introduces a lightweight communication mechanism via a posterior sparse autoencoder (SAE) trained to reconstruct activations from a frozen Vision Wormhole. Using a sparse payload format composed of uint16 indices and float16 values, the method achieves a 128× compression ratio with only 50 active features out of a 4096-dimensional dictionary, incurring an average accuracy drop of merely 0.08% across seven tasks on nine inference benchmarks. Further analysis via Jaccard similarity reveals substantial overlap in task-shared features, underscoring the efficiency of the proposed sparse representation.

Abstract

Latent-space communication allows heterogeneous vision-language model agents to exchange continuous representations without serializing visual and reasoning states into text. Vision Wormhole realizes this approach by translating visual features into a universal latent representation that can be consumed by another model, but every message is transported as a dense tensor of the same size regardless of its content. A fixed-capacity dense tensor therefore need not have a fixed effective information density: some messages may use only a small fraction of the available representational degrees of freedom. This observation suggests that the communication channel may be substantially compressible. We study its redundancy by fitting a post-hoc sparse autoencoder to frozen Vision Wormhole activations and measuring reconstruction, downstream utility, feature reuse, and token-level interventions across nine reasoning benchmarks. Relative to the original float32 transport, a uint16-index/float16-value sparse payload with k=4 active coefficients per token reduces the transmitted bytes by 128x. In a single-run evaluation, the seven-task non-AIME mean accuracy changes from 49.85% to 49.77%. The fitted 4096-element dictionary uses only 50 features, and task-level active sets have a mean pairwise Jaccard similarity of 0.906. These measurements establish strong post-hoc compressibility relative to the original transport, but do not yet isolate the incremental contribution of sparse coding from position selection, reduced precision, low-rank structure, or SAE optimization effects. The results motivate matched-payload comparisons and communication mechanisms whose payload adapts to the information used by each message.

Problem

• latent communication
• vision-language models
• sparse coding
• information redundancy
• post-hoc compression

Innovation

• post-hoc sparse coding
• latent communication
• vision-language models
• sparse autoencoder
• communication compression