Hera: Learning Long-Horizon Coordination for Device-Cloud Collaborative LLM Agents

2026-05-23

AI Summary

This work addresses the limitations of existing device-cloud large language model (LLM) collaboration approaches, which rely on coarse-grained, task-level routing and struggle to adapt to dynamically varying reasoning demands in long-horizon, multi-step tasks. To overcome this, we propose Hera, a step-level coordination framework that introduces a fine-grained dynamic routing mechanism. Hera achieves Pareto optimality between performance and cost through a two-stage training paradigm: an initial imitation learning phase with trajectory replay for cold-start initialization, followed by a reinforcement learning phase that integrates state clustering with joint reward-cost optimization. Experiments demonstrate that Hera attains 92.5% of the success rate of full cloud-based LLM execution on ALFWorld, WebShop, and AppWorld benchmarks while using only 46.3% of the cloud invocation steps, substantially outperforming current methods.

Abstract

Large language model (LLM) agents excel at solving complex long-horizon tasks through autonomous interaction with environments. However, their real-world deployment faces a fundamental device--cloud dilemma: on-device models are efficient but often brittle, while cloud models are stronger but costly in computation. State-of-the-art LLM device--cloud routers usually make coarse task-level decisions, which cannot adapt to the changing difficulty of multi-step agent interactions. To address this issue, we present Hera, a step-level device--cloud LLM agent coordinator for long-horizon tasks achieving a strong performance--cost Pareto frontier. Hera adopts a novel two-stage training paradigm: (1) imitation learning for cold-start, followed by (2) reinforcement learning that jointly optimizes task success and cloud usage efficiency. The first stage casts step-level routing as a supervised classification problem: the device agent is replayed on cloud trajectories, with each state labeled by the agreement between device and cloud actions. In the second stage, we perform cost-aware reinforcement learning by grouping identical states across trajectories and updating Hera with labels favoring higher expected return and fewer future cloud calls. We evaluate Hera on ALFWorld, WebShop, and AppWorld, where it consistently outperforms prior methods, achieving 92.5% of the cloud-only success rate with cloud use in only 46.3% of steps.

Problem

• device-cloud collaboration
• long-horizon tasks
• LLM agents
• step-level coordination
• cost-performance trade-off

Innovation

• step-level routing
• device-cloud collaboration
• two-stage training
• cost-aware reinforcement learning
• LLM agents