Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

2026-07-04

AI Summary

This work addresses the challenge that large language model agents struggle to efficiently leverage failed trajectories during long-horizon interactions, where full re-execution is costly and experience retrieval often dilutes critical signals. To overcome this, the authors propose PivoARL, a novel framework introducing a pivotal-turnpoint-based local retry mechanism and a criticality-aware credit assignment method. By conducting structured self-reflection to identify error boundaries, PivoARL retries only from key erroneous states while reusing correct trajectory prefixes to minimize redundant interactions. Integrating implicit reflection-based reward optimization with contrastive preference signals, the approach achieves significant performance gains across four agent tasks and seven question-answering benchmarks—improving average Pass@2/3 by 11.5%, enhancing Pass@1 on over 80% of tasks, and surpassing GiGPO by more than 45% in win rate on Minesweeper while reducing interaction steps by 42%.

Abstract

Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals. To address this, we propose PivoARL, a self-feedback retry framework for experience exploitation in LLM agents. PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions. From an information-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state-agnostic experience utilization. Based on this insight, we design a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns. We conduct a systematic evaluation on 4 agent tasks and 7 search-based QA benchmarks. Results show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5\% over MetaRL. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also consistently improves Pass@1 on over 80\% of the tasks. On Minesweeper environment, PivoARL improves over GiGPO by more than 45\% and reduces interaction turns by about 42\% on average compared with full-retry methods. Code is available at https://github.com/yuki-younai/PivoARL.

Problem

• LLM agents
• failed trajectories
• experience utilization
• signal dilution
• interaction cost

Innovation

• pivotal-aware retry
• self-feedback
• credit assignment
• experience exploitation
• LLM agents