The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

2026-02-17

AI Summary

This work addresses the high communication overhead and information loss inherent in discrete text-based messaging in multi-agent systems, as well as the limited scalability of existing latent-space approaches to heterogeneous models. The authors propose Vision Wormhole, a novel framework that leverages the vision encoder of a vision-language model (VLM) as a universal communication interface, enabling text-free coordination through a shared continuous latent space. By adopting a star topology, the framework reduces alignment complexity from O(N²) to O(N) and incorporates a label-free teacher-student distillation mechanism to align visual and textual reasoning pathways. Experiments across heterogeneous models—including Qwen-VL and Gemma—demonstrate that Vision Wormhole substantially lowers end-to-end communication latency while preserving reasoning fidelity comparable to conventional text-based communication.

Technology Category

• Computer Vision: Large Vision Models
• Multiagent Systems: Agent Communication
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems

Abstract

Multi-Agent Systems (MAS) powered by Large Language Models have unlocked advanced collaborative reasoning, yet they remain shackled by the inefficiency of discrete text communication, which imposes significant runtime overhead and information quantization loss. While latent state transfer offers a high-bandwidth alternative, existing approaches either assume homogeneous sender-receiver architectures or rely on pair-specific learned translators, limiting scalability and modularity across diverse model families with disjoint manifolds. In this work, we propose the Vision Wormhole, a novel framework that repurposes the visual interface of Vision-Language Models (VLMs) to enable model-agnostic, text-free communication. By introducing a Universal Visual Codec, we map heterogeneous reasoning traces into a shared continuous latent space and inject them directly into the receiver's visual pathway, effectively treating the vision encoder as a universal port for inter-agent telepathy. Our framework adopts a hub-and-spoke topology to reduce pairwise alignment complexity from O(N^2) to O(N) and leverages a label-free, teacher-student distillation objective to align the high-speed visual channel with the robust reasoning patterns of the text pathway. Extensive experiments across heterogeneous model families (e.g., Qwen-VL, Gemma) demonstrate that the Vision Wormhole reduces end-to-end wall-clock time in controlled comparisons while maintaining reasoning fidelity comparable to standard text-based MAS. Code is available at https://github.com/xz-liu/heterogeneous-latent-mas

Problem

• Multi-Agent Systems
• Latent-Space Communication
• Heterogeneous Agents
• Text-Free Communication
• Information Quantization Loss

Innovation

• latent-space communication
• heterogeneous multi-agent systems
• vision-language models
• universal visual codec
• model-agnostic communication