Closing the Feedback Loop: From Experience Extraction to Insight Governance in Verbal Reinforcement Learning

2026-06-16

AI Summary

This work addresses the vulnerability of large language model (LLM) agents in non-stationary environments to negative transfer and catastrophic forgetting caused by outdated experiences. To mitigate these issues, the authors propose a training-free, verbal reinforcement learning framework featuring a three-tiered architecture—rules, evidence, and skills—coupled with a feedback-driven curation loop that enables dynamic governance of experience into actionable insights. The core innovations include outcome-driven evaluation, structured evidence retention, a non-monotonic knowledge lifecycle, and compositional governance mechanisms, collectively offering the first systematic solution to the lack of insight governance in this paradigm. Empirical evaluation on financial forecasting tasks demonstrates that incorporating the curation loop significantly improves prediction accuracy and risk-adjusted returns, substantially outperforming both ablated variants without the loop and zero-shot baselines.

Abstract

Training-free verbal reinforcement learning enables LLM agents to learn from world feedback -- objective signals such as dynamic task outcomes, market returns, or demand forecasts -- by extracting verbal rules from experience and injecting them as context, updating the agent's behavior without parameter changes. However, in non-stationary environments these agents face a retention-forgetting dilemma: retaining stale insights causes negative transfer, while discarding them causes catastrophic forgetting when conditions recur. We identify four requirements for navigating this dilemma -- outcome-driven evaluation, persistent structured evidence, non-monotonic knowledge lifecycle, and compositional governance -- and show that existing methods invest heavily in experience extraction while underinvesting in insight governance. We propose a three-layer architecture -- rules, evidence, and skills -- connected by a feedback-driven curation loop that closes the governance gap. Rules capture distilled experience from world outcomes; evidence logs track each rule's reliability across episodes; skills govern which rules to apply, how to resolve conflicts, and when to abstain. On financial forecasting as a case study, where world feedback is naturally abundant, noisy, and non-stationary, we show that the same accumulated experience either degrades performance below the zero-shot baseline or dramatically improves accuracy and risk-adjusted returns, depending on whether the curation loop is present.

Problem

• verbal reinforcement learning
• non-stationary environments
• insight governance
• catastrophic forgetting
• experience extraction

Innovation

• verbal reinforcement learning
• insight governance
• non-stationary environments
• feedback-driven curation
• knowledge lifecycle