MARS: Efficient, Adaptive Co-Scheduling for Heterogeneous Agentic Systems

2026-04-14

AI Summary

This work addresses the GPU-CPU coupled resource scheduling challenges arising from multi-turn LLM-tool interactions in heterogeneous multi-agent systems. To tackle this, the authors propose MARS, a system featuring a unified information-flow architecture that enables global coordinated scheduling by decoupling admission control from execution. MARS incorporates an external control plane to prevent overload and an agent-centric internal scheduler that optimizes critical execution paths. Furthermore, it dynamically manages KV cache based on latency sensitivity. Experimental results demonstrate that MARS reduces end-to-end latency by up to 5.94×, achieves near-theoretical-maximum system throughput, and accelerates task completion time by 1.87× within the OpenHands framework.

Abstract

Large language models (LLMs) are increasingly deployed as the execution core of autonomous agents rather than as standalone text generators. Agentic workloads induce a temporal shift from single-turn inference to multi-turn LLM-tool loops, and a spatial shift from chat-scale, GPU-only execution to repository-scale, GPU-CPU co-located execution. Consequently, coordinating heterogeneous resource demands of agentic execution has emerged as a critical system challenge. We design and implement MARS, an efficient and adaptive co-scheduling system that globally coordinates heterogeneous agentic workloads under coupled GPU-CPU resource pressure. By establishing holistic visibility across GPU inference and CPU tool execution via a unified information stream, an external control plane in MARS decouples admission from execution to prevent heterogeneous resource oversubscription. An internal agent-centric scheduler further minimizes the end-to-end critical path by prioritizing latency-sensitive continuations and adaptively retaining KV cache state only when warm resumption yields a latency benefit. Our evaluations show that MARS reduces end-to-end latency by up to 5.94x while maintaining nearly maximal system throughput. We further integrate MARS as the serving backend for the OpenHands coding agent framework, demonstrating its real-world effectiveness by accelerating end-to-end task completion time by up to 1.87x. Our source code will be publicly available soon.

Problem

• heterogeneous agentic systems
• GPU-CPU co-scheduling
• resource coordination
• multi-turn LLM-tool loops
• agentic workloads

Innovation

• co-scheduling
• heterogeneous agentic systems
• KV cache management
• GPU-CPU coordination
• adaptive scheduling