Generative Semantic Communication via Textual Prompts: Latency Performance Tradeoffs

2024-09-15

arXiv.org

AI Summary

This work addresses ultra-low-bitrate semantic communication leveraging pre-trained multimodal/visual-language models (M/VLMs) on wireless edge and end devices, focusing on jointly optimizing text prompt generation, computation offloading decisions, and wireless/computational resource allocation to balance end-to-end latency and semantic fidelity under stringent bitrate constraints. Method: We propose a novel edge–end collaborative generative semantic communication framework and formulate the first joint optimization model for multi-user generative semantic communication—characterized by non-convexity and tightly coupled mixed-integer variables. To solve it, we design the Semantic–Latency–Joint (SLJ) matching algorithm, enabling Pareto-optimal trade-offs between latency and semantic quality. Contribution/Results: Experiments demonstrate that our approach significantly reduces end-to-end latency and improves semantic fidelity compared to conventional semantic-agnostic or non-collaborative baselines, while substantially expanding the Pareto performance frontier.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Learning to Search
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

This paper develops an edge-device collaborative Generative Semantic Communications (Gen SemCom) framework leveraging pre-trained Multi-modal/Vision Language Models (M/VLMs) for ultra-low-rate semantic communication via textual prompts. The proposed framework optimizes the use of M/VLMs on the wireless edge/device to generate high-fidelity textual prompts through visual captioning/question answering, which are then transmitted over a wireless channel for SemCom. Specifically, we develop a multi-user Gen SemCom framework using pre-trained M/VLMs, and formulate a joint optimization problem of prompt generation offloading, communication and computation resource allocation to minimize the latency and maximize the resulting semantic quality. Due to the nonconvex nature of the problem with highly coupled discrete and continuous variables, we decompose it as a two-level problem and propose a low-complexity swap/leaving/joining (SLJ)-based matching algorithm. Simulation results demonstrate significant performance improvements over the conventional semanticunaware/non-collaborative offloading benchmarks.

Problem

• Optimize edge-device collaboration for semantic communication
• Minimize latency in multi-user Gen SemCom framework
• Maximize semantic quality with resource allocation

Innovation

• Edge-device collaborative Gen SemCom framework
• Optimizes M/VLMs for semantic communication
• SLJ-based matching algorithm minimizes latency