Sema: Semantic Transport for Real-Time Multimodal Agents

2026-04-22

AI Summary

This work addresses the inefficiency of conventional network transmission mechanisms, which are optimized for human perception and thus incur high bandwidth overhead and latency for multimodal agents in real-time tasks, hindering effective semantic communication. The authors propose the first event-driven semantic transmission system tailored for intelligent agents, encoding audio into discrete semantic tokens and screen content into a hybrid representation combining accessibility trees or OCR-derived text with visual tokens. By employing a jitter-free burst transmission mechanism, the system prioritizes task-level semantic integrity over traditional signal fidelity. This approach marks the first shift in communication objectives from the Shannon–Weaver Layer A (signal) to Layer B (semantic). Evaluated under simulated wide-area network conditions, it achieves a 64-fold reduction in upstream audio bandwidth and 130–210× compression of screenshots, with task accuracy degradation of no more than 0.7 percentage points.

Technology Category

• Multiagent Systems: Agent Communication
• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Real-time multimodal agents transport raw audio and screenshots using networking stacks designed for human receivers, which optimize for perceptual fidelity and smooth playout. Yet agent models act as event-driven processors with no inherent sense of physical time, consuming task-relevant semantics rather than reconstructing signals in real time. This fundamental difference shifts the transport goal from the technical problem of signal fidelity (Shannon-Weaver Level A) to the semantic problem of meaning preservation (Level B). This mismatch imposes significant overhead. In visual pipelines, screenshot upload accounts for over 60% of end-to-end action latency on constrained uplinks, and in voice pipelines, conventional transport carries massive redundancy, sending 43-64x more data than needed to maintain task accuracy. We present Sema, a semantic transport system that combines discrete audio tokenizers with a hybrid screen representation (lossless accessibility-tree or OCR text, plus compact visual tokens) and bursty token delivery that eliminates jitter buffers. In simulations under emulated WAN conditions, Sema reduces uplink bandwidth by 64x for audio and 130-210x for screenshots while preserving task accuracy within 0.7 percentage points of the raw baseline.

Problem

• semantic transport
• multimodal agents
• bandwidth overhead
• meaning preservation
• real-time communication

Innovation

• semantic transport
• multimodal agents
• discrete tokenization
• bandwidth efficiency
• task-oriented communication