Learning from Environmental Feedback: Credit Assignment across Multiple Timescales for Agentic Reinforcement Learning

2026-08-08

AI Summary

This work addresses the challenge of sparse and delayed rewards in real-world reinforcement learning, which hinders accurate credit assignment to intermediate actions. The authors propose a multi-timescale credit assignment method that explicitly leverages naturally occurring process information from environment interactions. By modeling short-term action effects, identifying medium-term ineffective interactions, and reweighting trajectory-level returns, the approach constructs fine-grained supervisory signals across short, medium, and long temporal scales. Evaluated on the ALFWorld and WebShop benchmarks, the method significantly improves both task success rates and completion quality compared to strong baselines, demonstrating the effectiveness and novelty of process-informed, multi-timescale credit assignment.

Abstract

Agentic reinforcement learning (RL) often suffers from delayed and sparse rewards in real-world environments. A promising solution to this challenge is credit assignment, which aims to decompose trajectory-level rewards and provide more fine-grained supervision for intermediate decisions. However, existing credit assignment approaches ignore the rich process information naturally generated during environment interaction, e.g., interaction history. We argue that such information provides valuable supervision for identifying the contribution of individual actions. To this end, we propose Environmental Feedback-based Credit Assignment (EFCA), a multi-timescale credit assignment approach for long-horizon agentic RL. EFCA complements the long-term outcome signal with two environment-grounded process signals: a short-term feedback signal that captures the immediate effect of the current action and a medium-term state-history signal that identifies ineffective patterns from recent interactions. Both signals are directly extracted from environment feedback and integrated through a return reweighting mechanism. Experiments on ALFWorld and WebShop demonstrate that EFCA consistently improves both task success and task quality over strong baselines, highlighting the effectiveness of environment-grounded multi-timescale credit assignment for long-horizon agentic RL.

Problem

• credit assignment
• reinforcement learning
• delayed rewards
• sparse rewards
• multi-timescale

Innovation

• credit assignment
• multi-timescale learning
• environmental feedback
• agentic reinforcement learning
• return reweighting