Token-Oriented Semantic Communication with Pretrained Vision Transformers

2026-08-26

AI Summary

为解决边缘系统中传输token嵌入的通信成本高和互操作性差的问题，提出了一种面向token的语义通信框架，通过选择性传输与任务相关的压缩图像潜变量来实现细粒度传输。

Abstract

Token communications realize the semantic communication principle at the granularity of transformer tokens, providing a promising direction for client--server collaborative inference in resource-constrained edge systems. However, directly transmitting token embeddings presents two practical challenges: substantial communication cost and limited interoperability across model-specific token embedding spaces. To address these challenges, we propose a \emph{token-oriented} semantic communication framework. In this framework, token-level task relevance determines which compressed image latents are transmitted, enabling token-granular transmission without directly transmitting token embeddings. The framework is modular, coordinating three pretrained components---a lightweight client-side vision transformer (ViT), a learned image compression (LIC) model, and a large server-side ViT---without end-to-end training. The key enabler is the one-to-one spatial alignment between ViT patch tokens and the LIC latent vectors, which allows token-level task relevance to directly determine which latent vectors are transmitted. Building on this alignment, token-aligned LIC selectively transmits task-relevant latents, layer-selective attention rollout estimates token relevance from a selected range of attention layers in a single forward pass, and surrogate token substitution adapts the frozen server model by optimizing a single learnable token. Experiments on ImageNet show that the proposed framework achieves a more favorable rate--accuracy trade-off than recent semantic communication schemes, hand-crafted codecs, and task-agnostic LIC models.

Problem

• semantic communication
• transformer tokens
• communication cost
• interoperability
• resource-constrained edge systems

Innovation

• token-oriented semantic communication
• transformer tokens
• learned image compression (LIC)
• spatial alignment
• layer-selective attention rollout