Retrospective Progress-Aware Self-Refinement for LLM Agent Training

2026-06-12

AI Summary

This work addresses the limited metacognitive awareness of task progress in existing large language model (LLM) agents within reinforcement learning settings, which hinders their performance on long-horizon tasks. To overcome this limitation, the authors propose the RePro framework, which introduces a novel retrospective progress-aware mechanism: after forward execution, the agent retrospectively analyzes its trajectory, self-evaluates the contribution of each step toward task completion, and generates intrinsic progress signals. Coupled with Retrospection Warmup for initialization and the RePro-PO policy optimization algorithm, the approach enables effective training with minimal external demonstrations. Experimental results demonstrate that RePro significantly enhances the performance of Qwen-series models across WebShop, ALFWorld, and Sokoban benchmarks, yielding up to a 12% absolute improvement in task success rate.

Abstract

LLM-based agents trained with reinforcement learning optimize step-wise action prediction but lack metacognitive awareness of task progress, inducing a gap that hinders long-horizon scaling. A pilot study reveals that online progress prompting hurts performance while retrospective demonstrations help, yet this capability cannot emerge from outcome-reward training alone. We present RePro, Retrospective Progress-Aware Training, a framework that trains agents to self-generate progress signals via a forward-then-reflect rollout paradigm: the agent executes actions online, then retrospectively reassesses its step-wise progress given the completed trajectory and known outcome. RePro initializes with a Retrospection Warmup that teaches reflection format from minimal external demonstrations, then further trains through RePro-PO with a composite reward that produces self-generated signals without continuous external supervision. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to $12\%$ absolute success rate gains.

Problem

• LLM agent
• metacognitive awareness
• task progress
• long-horizon reasoning
• reinforcement learning

Innovation

• Retrospective Self-Refinement
• Progress Awareness
• LLM Agent Training
• Metacognitive Reflection
• RePro-PO