Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence

2026-07-08

AI Summary

This work addresses the challenge of deploying large language models (LLMs) on resource-constrained edge devices, where centralized inference suffers from high latency and privacy concerns. To overcome these limitations, we propose the first elastic multi-device collaborative LLM inference framework tailored for real-world edge environments. By integrating dynamic scheduling, model partitioning, and a lightweight cooperative execution mechanism, our approach transcends the resource constraints of individual devices and enables elastic scaling of model size. Experimental results demonstrate that the proposed framework not only meets user quality-of-service (QoS) requirements but also achieves up to a 16.5% improvement in inference accuracy compared to the best existing single-device solutions.

Abstract

Large language models (LLMs) are widely used in intelligent services due to their remarkable capability in generative tasks. Typically, LLM-based services process the inference requests of the users in a centralized data center. Unfortunately, such centralized execution has limitations for end-users, such as increased response latency with communication overhead and privacy leakage risk. To alleviate the aforementioned limitations, there have been increasing pushes to execute LLM inference locally on user-end devices. However, the limited resources of a single edge device impose restrictions on achievable accuracy of LLMs. To overcome the issue, we first propose to leverage multiple user-end devices available at the edge for LLM inference, enabling the execution of larger models. Specifically, we propose Voltron, a novel on-device LLM inference framework that elastically utilizes multiple user-end devices for LLM inference execution while adapting to diverse real-world edge environments. In our evaluation, Voltron achieves up to 16.5% higher accuracy than state-of-the-art LLMs that can be executed on a single edge device, satisfying user QoS requirements.

Problem

• LLM inference
• edge intelligence
• resource constraints
• multi-device execution
• latency and privacy

Innovation

• elastic multi-device execution
• on-device LLM inference
• edge intelligence
• resource-constrained devices
• distributed inference