Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning

2026-06-18

AI Summary

This work addresses the open challenge of enabling large language models (LLMs) to continuously learn, accumulate experience, and dynamically update their environmental understanding over extended lifetimes. We propose the “Connectors of Dynamics” (CoD) framework, which formalizes continual learning and context self-updating as trainable meta-capabilities. Through end-to-end reinforcement learning, the framework alternates between task execution and context refinement across long-horizon tasks. To support this paradigm, we design specialized tasks and cross-domain evaluation environments, and develop a training infrastructure based on GRPO-style algorithms with fine-grained credit assignment, enabling long-trajectory replay and iterative context updates. Experiments demonstrate that our approach effectively trains LLM agents with strong cross-domain generalization, achieving consistent performance gains within training domains, across unseen domains, and in Ralph-loop settings.

Abstract

This work presents a general framework for training large language models (LLMs) to "Connect the Dots" (CoD), a meta-capability required by long-lifecycle agents: as an LLM-based AI agent gets deployed in an environment, it solves a long sequence of tasks while continuously exploring the environment, learning from its own experiences, and iteratively self-updating its context about the environment, thereby achieving progressively better performance on future tasks conditioned on the updated context. Major components of the CoD framework include: (1) algorithm design and infrastructure for end-to-end reinforcement learning (RL) with long rollout sequences interleaving solve-task and update-context episodes; (2) tasks and environments for incentivizing and eliciting the targeted meta-capability in LLMs during training, as well as for faithfully measuring progress during evaluation. We present proof-of-concept implementations of the CoD framework, including a GRPO-style RL algorithm with fine-grained credit assignment, as well as tasks and environments tailored to the targeted meta-capability (rather than domain-specific LLM capabilities or standard task-by-task RL). Empirical results validate the efficacy of end-to-end RL training in the CoD setting, and demonstrate the potential for out-of-distribution generalization -- within the training domains, across different domains, and from CoD to Ralph-loop settings -- of the elicited meta-capability. Our investigation of CoD connects several lines of prior works, and opens up new opportunities for advancing LLMs and AI agents. To facilitate further research and applications, we release our implementations at \url{https://github.com/agentscope-ai/Trinity-RFT/tree/research/cod/examples/research_cod}.

Problem

• long-lifecycle agents
• cross-domain generalization
• reinforcement learning
• large language models
• meta-capability

Innovation

• Connect the Dots
• long-lifecycle agents
• cross-domain generalization
• end-to-end reinforcement learning
• meta-capability