Training Language Agents to Learn from Experience

2026-05-19

AI Summary

Existing language agents are limited to within-task self-correction and struggle to convert experience into generalizable knowledge. This work proposes an In-context Training (ICT) framework, wherein a reflection model generates system prompts from execution trajectories to guide an executor model toward improved performance on novel tasks. Crucially, ICT employs unsupervised reinforcement learning to directly train the reflection capability from raw experience, without relying on external supervision. This approach represents the first demonstration of cross-task self-improvement in language agents, substantiating that “learning from experience” can be acquired through training. Evaluated on ALFWorld and MiniHack benchmarks, ICT significantly outperforms baseline methods and exhibits strong generalization—even to unseen task families and substantially different environments.

Abstract

Language agents can adapt from experience in interactive environments, but current reflection-based methods can only self-correct within a single task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in language agents. In ICT, a reflector model observes trajectories collected by an actor model and generates system prompts intended to improve the actor's performance on future unseen tasks. We then propose an RL-based training pipeline for learning such reflections directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform an untrained baseline on most held-out task families, showing that the ability to learn from experience can itself be learned. In some cases, we observe generalisation beyond the benchmark on which the reflector was trained, to substantially different environments. Finally, we introduce MetaGym, a generic Python library for constructing meta-environments, enabling future research on self-improving language agents.

Problem

• language agents
• self-improvement
• cross-task learning
• experience distillation
• reflection

Innovation

• In-context Training
• language agents
• self-improvement
• reinforcement learning
• cross-task generalization