Next-Generation Agentic Reinforcement Learning Systems Enable Self-Evolving Agents

2026-07-01

AI Summary

This work addresses the challenge of capability stagnation in enterprise-deployed large language model agents, which typically lack mechanisms for continuous learning. To overcome this limitation, we propose AReaL2.0—an online reinforcement learning system enabling agent self-evolution through three core components: a fine-grained standardized trajectory protocol, a workload-to-learning-data translation proxy, and a statistically driven evolution control plane. Integrated with an observability stack, the system supports online policy weight updates and co-evolution of contextual mechanisms. Experimental results demonstrate that the architecture effectively leverages real-world deployment workloads to drive continuous self-optimization of agents, thereby validating the feasibility and efficacy of self-evolving agents in enterprise environments.

Abstract

LLM agents are rapidly being deployed in production, including coding assistants, customer-support chatbots, and scientific research assistants, yet they remain fundamentally static in enterprise deployment. The LLM weights, system prompts, tool repertoires, and in-context harnesses are frozen at deployment time, and any improvement requires a manual loop of human-curated data collection, offline fine-tuning, modification of the agentic paradigm, and re-deployment. Recent work on self-evolving agents, such as OpenClaw for individual users, indicates that the next leap in agent capability will come from agents that continually learn from their own experience. In this paper, we argue that this vision for self-evolving agent deployment is being held back for enterprise-level large-scale agentic service not by reinforcement learning (RL) algorithms but by agentic online RL systems. Specifically, current agentic RL systems and the surrounding observability software stack are inadequate along three essential aspects: (i) there is no standardized agent trajectory data protocol capable of carrying RL learning signals at step granularity across heterogeneous agent paradigms; (ii) there is no enterprise-grade comprehensive data proxy that converts real workloads into governed learning substrates; and (iii) there is no unified agent evolution control plane that automatically decides, based on trajectory statistics, when to update policy weights or evolve the in-context harness. The next generation of agentic RL systems must be co-designed around these three pillars, and we sketch concrete architectures, case studies, and counter-arguments. We instantiate one branch through AReaL2.0, reorganizing existing RL infrastructure into an agent-oriented online RL loop for policy weight updates from deployed workloads.

Problem

• agentic reinforcement learning
• self-evolving agents
• online RL systems
• enterprise deployment
• trajectory data protocol

Innovation

• self-evolving agents
• agentic reinforcement learning
• trajectory data protocol
• online RL systems
• agent evolution control plane