HeatCache: Thermal-aware Energy-efficient LLM Inference Scheduling for Chassis-level Liquid Cooling in Sustainable Edge Server Rooms

2026-09-11

AI Summary

为解决边缘服务器多GPU推理产生的高能耗和热问题，提出HeatCache调度器，通过利用AIO液体冷却系统作为临时热缓冲区来优化能源使用，同时满足热安全性和服务级别目标。

Abstract

LLM inference is increasingly deployed at institution-scale edges to meet service requirements. However, multi-GPU inference consumes a large amount of electricity and produces substantial heat. To improve sustainability, operators and regulations often demand raising the ambient setpoint to reduce cooling electricity. This can increase thermal throttling and hardware aging, leading to Service-Level Objective violations. In this paper, we present HeatCache, a thermal-aware, energy-efficient LLM inference scheduler for commercial chassis-level AIO liquid-cooled GPUs at sustainable ambient temperatures. HeatCache treats AIO loops as a temporary heat buffer, measured by heat budget and schedules requests to minimize energy subject to thermal safety and SLO constraints, based on an electrical-informed heat-demand estimation from HeatiTS. We implement HeatCache atop vLLM and show that it reduces computing energy by up to 18.0%, decreases thermal-throttle exposure by 81.7%, and maintains SLO violation rates below 0.9% even up to $48~^{\circ}\mathrm{C}$.

Problem

• LLM inference
• sustainable edge server rooms
• thermal throttling
• hardware aging
• Service-Level Objective

Innovation

• Thermal-aware
• Energy-efficient
• LLM Inference Scheduling
• AIO Liquid Cooling
• Sustainable Ambient Temperatures