DRLM: Deep Reinforcement Learning-Based LLM Query Orchestration in Edge Environments

2026-08-31

AI Summary

本文提出DRLM框架，利用深度强化学习解决边缘环境中大语言模型查询调度问题，通过预测器和PPO代理优化决策，减少延迟并保持准确性。

Abstract

Large language model (LLM) services increasingly process heterogeneous queries with diverse latency, accuracy, and resource requirements. While edge deployment reduces response time, the heterogeneity of devices and the diversity of model families, parameter scales, and quantization levels make efficient LLM query orchestration challenging. This paper introduces DRLM, a Deep Reinforcement Learning-based LLM query orchestration framework in edge environments. DRLM integrates two lightweight predictors: (i) a class-conditioned quality estimator that maps queries to semantic categories and infers model performance, and (ii) a feature-driven latency predictor that estimates inference time across model-device configurations. These predictions, combined with system state, feed a factorized Proximal Policy Optimization (PPO) agent that performs state-aware orchestration decisions. To enable data-driven orchestration, we construct a large-scale benchmarking dataset with 223 835 measurements spanning 1258 queries, 6 query classes, 8 model families (32 deployed instances), 5 quantization levels, and heterogeneous edge devices. Evaluation on a 64-node edge cluster and comparison with three baselines and two state-of-the-art methods show that DRLM reduces inference latency by up to 51% and queuing delay by up to 67 %, while incurring at most 8% accuracy loss. It improves latency under increasing workloads up to 61.4%, demonstrating robust and stable orchestration.

Problem

• LLM Query Orchestration
• Edge Environments
• Heterogeneous Queries
• Model Families
• Quantization Levels

Innovation

• Deep Reinforcement Learning
• Query Orchestration
• Edge Environments
• Latency Reduction
• PPO Agent