KAIROS: Stateful, Context-Aware Power-Efficient Agentic Inference Serving

2026-04-17

AI Summary

Existing power management approaches struggle to handle the long-lived, contextually evolving multi-turn interaction workloads characteristic of AI agent inference, often triggering memory thrashing that degrades both performance and energy efficiency. This work proposes a novel, fine-grained, thrash-aware co-management mechanism that integrates agent contextual state into power control for the first time. By combining agent-granularity request tracking, context-aware local regulation, cross-instance agent routing, and joint optimization of GPU frequency and concurrency, the approach significantly improves energy efficiency while meeting performance targets. Experimental results across diverse software and data engineering agent tasks demonstrate an average power reduction of 27%, with improvements reaching up to 39.8%.

Technology Category

• Machine Learning: Hardware-aware ML
• Multiagent Systems: Mechanism Design
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Power has become a central bottleneck for AI inference. This problem is becoming more urgent as agentic AI emerges as a major workload class, yet prior power-management techniques focus almost entirely on single-turn LLM serving. Our analysis shows that agentic serving behaves fundamentally differently: each request carries long-lived context that evolves across tool-interleaved turns, and lowering GPU frequency can push the system into a thrashing regime where memory pressure sharply worsens both performance and power efficiency. These observations show that power optimization for agentic serving requires rethinking. We present KAIROS, a context-aware power optimization system for agentic AI serving. KAIROS uses agent context as a first-class control signal to jointly manage GPU frequency, per-instance concurrency, and multi-instance request placement. This enables KAIROS to save power when memory headroom exists while avoiding thrashing and preserving performance targets. At a high level, KAIROS tracks requests at agent granularity, adapts local control to context growth and agent progress, and routes agents across instances to jointly improve power efficiency and memory stability. Evaluated across diverse software and data engineering agentic tasks, KAIROS achieves an average of 27% (up to 39.8%) power reduction while meeting the performance targets.

Problem

• agentic AI
• power efficiency
• inference serving
• context-aware
• GPU frequency

Innovation

• agentic inference
• context-aware power management
• GPU frequency scaling
• memory thrashing avoidance
• multi-instance scheduling