Strait: Perceiving Priority and Interference in ML Inference Serving

2026-04-30

AI Summary

This work addresses the challenge of simultaneously supporting task prioritization and accurate latency prediction in machine learning inference services under high GPU utilization, a scenario where existing systems often fail to meet deadline guarantees. To this end, the authors propose a scheduling system tailored for dual-priority traffic that explicitly models contention in data transfers and interference among kernel executions. Building upon this model, they develop an adaptive, interference-aware latency predictor and integrate it into a priority-aware scheduling mechanism. Experimental results demonstrate that, under high load, the proposed approach reduces deadline violation rates for high-priority tasks by 1.02–11.18 percentage points while maintaining acceptable performance degradation for low-priority tasks. Furthermore, it achieves better fairness compared to software-defined preemption schemes.

Abstract

Machine learning (ML) inference serving systems host deep neural network (DNN) models and schedule incoming inference requests across deployed GPUs. However, limited support for task prioritization and insufficient latency estimation under concurrent execution may restrict their applicability in on-premises scenarios. We present \emph{Strait}, a serving system designed to enhance deadline satisfaction for dual-priority inference traffic under high GPU utilization. To improve latency estimation, Strait models potential contention during data transfer and accounts for kernel execution interference through an adaptive prediction model. By drawing on these predictions, it performs priority-aware scheduling to deliver differentiated handling. Evaluation results under intense workloads suggest that Strait reduces deadline violations for high-priority tasks by 1.02 to 11.18 percentage points while incurring acceptable costs on low-priority tasks. Compared to software-defined preemption approaches, Strait also exhibits more equitable performance.

Problem

• ML inference serving
• task prioritization
• latency estimation
• deadline satisfaction
• GPU interference

Innovation

• priority-aware scheduling
• latency prediction
• GPU interference modeling
• ML inference serving
• deadline satisfaction