Generation Quality-Latency Tradeoff-Aware Inference Offloading for Multimodal LLMs in Cloud-Edge Continuum

2026-06-13

AI Summary

This work addresses the challenge of balancing generation quality and response latency in multimodal large language model (MLLM) inference within cloud-edge collaborative environments. To this end, the authors propose QLMIO, a novel framework that enables the first joint quality–latency-aware inference offloading decision-making mechanism. They also introduce MIOBench, the first real-world benchmark dataset tailored for this task. By integrating a quality–latency trade-off-aware offloading strategy, a cloud-edge协同 architecture, and a customized performance prediction model, QLMIO achieves task completion rates comparable to pure cloud-based inference while reducing inference latency by up to 58.14%.

Abstract

Beyond pure cloud, some efforts are being made to deploy Large Language Models (LLMs) in edge to accelerate inference response. So the deployment of LLMs in cloud-edge continuum becomes a promising paradigm, where the tasks involving multimodal data occupy a large part of requests. Under this continuum, users usually concern about multiple Quality-of-Service (QoS) attributes, but it is always intractable to jointly optimize them. In this paper, we propose to study the joint optimization of those attributes and focus on two key representatives, i.e., content generation quality and response latency. We propose to study the offloading technology to achieve a tradeoff between the two objectives in the cloud-edge collaborative Multimodal LLM (MLLM) system. However, it is highly difficult to predict generation quality and inference latency for MLLM inference tasks while optimizing this offloading process. To address these unprecedented difficulties, we propose a Quality-Latency Tradeoff-Aware MLLM Inference Offloading (QLMIO) framework to make decisions that optimally balance generation quality and response latency. Meanwhile, recognizing the absence of publicly available datasets tailored to the MLLM inference offloading problem, we constructed a real-world cloud-edge collaborative MLLM system and subsequently collected an MLLM Inference Offloading Benchmark (MIOBench) to comprehensively evaluate our framework and facilitate the study of this problem. Extensive experimental results demonstrate that the QLMIO framework reduces latency by up to 58.14\% compared to baselines, while simultaneously matching the task completion rate achieved under the case that executes all requests exclusively on a cloud server. The dataset and codes are available at Github.

Problem

• Multimodal LLMs
• Inference Offloading
• Quality-Latency Tradeoff
• Cloud-Edge Continuum
• QoS Optimization

Innovation

• Inference Offloading
• Multimodal LLMs
• Quality-Latency Tradeoff
• Cloud-Edge Continuum
• MIOBench