GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization

2026-05-12

AI Summary

This work addresses the critical gap in existing large language model (LLM) request routing strategies, which typically overlook carbon emissions and energy sustainability, thereby failing to jointly optimize response quality, latency, and environmental impact across heterogeneous model pools. To bridge this gap, we propose GAR, a green-aware routing framework that, for the first time, integrates real-time grid carbon intensity and model-specific energy consumption into routing decisions. GAR employs constrained multi-objective optimization to minimize inference-related carbon emissions while satisfying minimum accuracy and p95 latency service-level constraints. The framework combines lightweight performance and carbon estimators, adaptive constraint tuning, and an online primal-dual algorithm (GAR-PD). Evaluated on heterogeneous LLM pools ranging from 7B to 70B parameters and standard NLP benchmarks, GAR achieves substantial reductions in carbon emissions without compromising high accuracy or latency compliance.

Abstract

The growing deployment of large language models (LLMs) makes per-request routing essential for balancing response quality and computational cost across heterogeneous model pools. Current routing methods rarely consider sustainable energy use and CO2 emissions as optimization objectives, despite grid carbon intensity varying by time and region, and models differing significantly in energy consumption. To address this gap, we introduce Green-Aware Routing (GAR), a constrained multi-objective optimization framework that minimizes per-request CO2 emissions subject to explicit accuracy floors and p95-latency service-level objectives (SLOs). GAR employs adaptive constraint optimization through per-dataset floor tuning and incorporates lightweight estimators for correctness, tail latency, and carbon emissions, enabling real-time routing decisions without additional inference passes. We present GAR-PD, a practical online primal-dual routing algorithm for rolling carbon budgets, alongside heuristic variants that achieve high feasibility coverage while limiting accuracy degradation. Comprehensive experiments across standard NLP benchmarks with heterogeneous LLM pools (7B-70B) demonstrate that GAR achieves substantial carbon reductions while maintaining competitive accuracy and p95 latency guarantees, providing a practical, theoretically grounded approach to sustainable LLM inference.

Problem

• carbon-aware routing
• LLM inference
• CO2 emissions
• sustainable AI
• request routing

Innovation

• carbon-aware routing
• constrained optimization
• LLM inference
• green AI
• service-level objectives