EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning

2026-06-16

AI Summary

Traditional reinforcement learning struggles with sparse rewards in long-horizon tasks and fails to fully exploit environmental dynamics embedded in interaction trajectories. This work proposes EnvRL, a novel framework that, for the first time, incorporates environmental dynamics as an implicit supervisory signal into the reinforcement learning process of large language model agents. By jointly training with policy optimization algorithms such as GRPO, EnvRL leverages two auxiliary objectives—state prediction and inverse dynamics modeling—to enable agents to internalize the underlying mechanisms of the environment. Evaluated on the ALFWorld and WebShop benchmarks, the method significantly improves task success rates; for instance, the Qwen-2.5-1.5B-Instruct model achieves gains from 72.8% to 77.4% on ALFWorld and from 56.8% to 67.0% on WebShop.

Abstract

Reinforcement learning (RL) has emerged as a powerful paradigm for training Large Language Models (LLMs) as agents. However, conventional RL methods for long-horizon agentic tasks often struggle with sparse outcome rewards. Intuitively, this overlooks the rich environment dynamics information contained in rollout interaction trajectories. We argue that the interaction experience inherently serves as an implicit supervision signal, reveals the underlying transition mechanisms of the environment, and enables the agent to construct a more accurate internal model of the environment.. Therefore, in this work, we investigate how to leverage this additional signal to improve policy learning. Specifically, we propose EnvRL, a framework that incorporates environment dynamics learning into agentic RL via two auxiliary objectives: state prediction and inverse dynamics. By jointly optimizing with the primary RL objective, we encourage the agent to internalize environment dynamics from its own interaction experience. Extensive experiments on two long-horizon agentic benchmarks demonstrate that EnvRL achieves significant improvements on success-rates over RL-only baselines, e.g., when trained with GRPO, lifting Qwen-2.5-1.5B-Instruct from 72.8% to 77.4% on ALFWorld, and from 56.8% to 67.0% on WebShop.

Problem

• reinforcement learning
• sparse rewards
• environment dynamics
• long-horizon tasks
• agentic learning

Innovation

• environment dynamics
• agentic reinforcement learning
• state prediction
• inverse dynamics
• auxiliary objectives