A Kubernetes-Native Request Router for Quality-Aware Inference Serving in the Computing Continuum

2026-09-17

AI Summary

研究提出ASRB，一种基于Kubernetes的请求路由机制，通过综合考虑基础设施信息、响应时间和质量指标来优化机器学习工作负载的服务，实现灵活的延迟-准确度权衡。

Abstract

We introduce Adaptive Score-based Routing Balancer (ASRB), a dynamic, score-based request routing mechanism for Kubernetes-based service deployments over the computing continuum. ASRB jointly considers infrastructure-level information, response time measurements, and application-level quality indicators, with a particular focus on serving Machine Learning (ML) workloads. For these workloads, ASRB balances requests over service instances deployed in the continuum, following service provider-defined policies encoded as weighted combinations of QoS criteria to flexibly address latency-accuracy trade-offs. To drive routing decisions and swiftly adapt to changes in the operating environment, ASRB monitors a range of runtime metrics across multiple system layers. To deal with the associated monitoring overhead, particularly important for large-scale deployments, it selectively and adaptively controls monitoring intensity without sacrificing on routing quality. ASRB is implemented without requiring any modifications to Kubernetes, making it straightforward to deploy and operate in existing cluster environments. Our testbed experiments demonstrate the versatility of ASRB: When tuned for latency reduction, it achieves at least 10 ms lower mean response time compared with latency-oriented state-of-the-art routing mechanisms, while it achieves higher accuracy when this is prioritized through specific configurations, thus enabling flexible and operator-controllable trade-offs. At the same time, it attains reduced failure rates, higher responsiveness to changes in the operating environment, and up to ~70% less monitoring cost than relevant state-of-the-art solutions, at the potential expense of only a modest latency penalty in some configurations.

Problem

• Kubernetes
• Request Routing
• Quality of Service
• Machine Learning Workloads
• Latency-Accuracy Trade-offs

Innovation

• Adaptive Score-based Routing Balancer (ASRB)
• Kubernetes
• Quality-Aware Inference Serving
• Latency-Accuracy Trade-offs
• Dynamic Request Routing