CoDA: A Context-Decoupled Hierarchical Agent with Reinforcement Learning

2025-12-14

AI Summary

To address the “context explosion” problem—caused by cumulative long-context dependencies in reinforcement learning–driven LLM agents during complex, multi-step tasks—this paper proposes PECO, a hierarchical decoupling architecture. Built upon a single LLM backbone, PECO strictly isolates the context spaces of the high-level Planner and low-level Executor. It introduces a role-aware context scheduling mechanism and establishes a trajectory-level end-to-end joint optimization framework, wherein policy updates explicitly depend on role-specific states. Crucially, PECO achieves the first hard isolation of planning and execution contexts while enabling their co-training. Evaluated on multi-hop question answering benchmarks, PECO significantly outperforms state-of-the-art methods and maintains robust performance under long-context conditions, whereas baseline models suffer severe degradation.

Technology Category

• Multiagent Systems: Multiagent Planning
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large Language Model (LLM) agents trained with reinforcement learning (RL) show great promise for solving complex, multi-step tasks. However, their performance is often crippled by "Context Explosion", where the accumulation of long text outputs overwhelms the model's context window and leads to reasoning failures. To address this, we introduce CoDA, a Context-Decoupled hierarchical Agent, a simple but effective reinforcement learning framework that decouples high-level planning from low-level execution. It employs a single, shared LLM backbone that learns to operate in two distinct, contextually isolated roles: a high-level Planner that decomposes tasks within a concise strategic context, and a low-level Executor that handles tool interactions in an ephemeral, isolated workspace. We train this unified agent end-to-end using PECO (Planner-Executor Co-Optimization), a reinforcement learning methodology that applies a trajectory-level reward to jointly optimize both roles, fostering seamless collaboration through context-dependent policy updates. Extensive experiments demonstrate that CoDA achieves significant performance improvements over state-of-the-art baselines on complex multi-hop question-answering benchmarks, and it exhibits strong robustness in long-context scenarios, maintaining stable performance while all other baselines suffer severe degradation, thus further validating the effectiveness of our hierarchical design in mitigating context overload.

Problem

• Addresses context explosion in RL-trained LLM agents
• Decouples high-level planning from low-level execution
• Mitigates performance degradation in long-context scenarios

Innovation

• Hierarchical agent decouples planning and execution
• Shared LLM backbone operates in two isolated roles
• PECO reinforcement learning jointly optimizes both roles