Energy-Efficient Multimodal Inference Serving with Tri-serve

2026-06-28

AI Summary

This work addresses three energy-efficiency bottlenecks in multimodal inference services on GPUs caused by hardware-managed dynamic frequency scaling: inter-stage pipeline stalls due to dependency-induced idling, an inverse correlation between arithmetic intensity and operating frequency, and thermal throttling triggered by high arithmetic intensity. To jointly mitigate these issues without requiring hardware modifications, the authors propose Tri-serve, the first software-defined dynamic voltage and frequency scaling (DVFS) controller that co-optimizes arithmetic intensity, thermal throttling, and pipeline scheduling through a unified analytical model. Experimental evaluation on commercial GPUs demonstrates that Tri-serve improves energy efficiency by 22% while strictly preserving inference latency and throughput.

Abstract

Multimodal model inference creates substantial energy demand with growing performance requirements. Within GPUs, power is autonomously managed by an on-board power management unit (PMU), which makes frequency boosting/throttling decisions. However, we find that these hardware-managed frequency decisions can cause significant power inefficiency. This work identifies three classes of power inefficiencies within modern multimodal inference serving: (1) inter-stage dependency stalls run at near maximum frequency despite being idle; (2) anti-correlation between auto-boost frequency and arithmetic intensity (A.I.) results in compute-bound phases (e.g., prefill) running at lower frequency and vice versa; and (3) thermal throttling degrades SM frequency and throughput. We propose Tri-serve, a software-based DVFS controller that jointly accounts for three classes of inefficiency -- inter-stage Dependency stalls, the Arithmetic-intensity effect on frequency and power, and the Thermal-throttling effect of high A.I. phases -- to deliver energy-efficient multimodal serving on commodity GPUs. We show that Tri-serve achieves 22% energy efficiency improvement with no latency or throughput impacts.

Problem

• energy efficiency
• multimodal inference
• GPU power management
• thermal throttling
• arithmetic intensity

Innovation

• multimodal inference
• energy efficiency
• DVFS
• arithmetic intensity
• thermal throttling