Re-ReST: Reflection-Reinforced Self-Training for Language Agents

2024-06-03

Conference on Empirical Methods in Natural Language Processing

AI Summary

Language agents’ self-training suffers from reliance on human annotations or strong teacher models, and often produces low-quality self-generated samples. Method: This paper proposes Re-ReST, the first framework to integrate reflective mechanisms into self-training—leveraging environmental feedback (e.g., unit test outcomes) for inference-time dynamic correction without ground-truth supervision. It introduces a reflector module for modeling reflection and a sample rewriting component to online refine outputs and efficiently construct high-quality datasets. Contribution/Results: Re-ReST generalizes across diverse tasks—including QA, decision-making, code generation, VQA, and text-to-image synthesis. On HotpotQA and AlfWorld, self-training alone improves baseline performance by 7.6% and 28.4%, respectively; Re-ReST further yields gains of 2.0% and 14.1%, demonstrating the effectiveness and broad applicability of reflection-enhanced self-training.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Finetuning language agents with reasoning-action trajectories is effective, but obtaining these trajectories from human annotations or stronger models is costly and sometimes impractical. In this paper, we investigate the use of self-training in language agents, which can generate supervision from the agent itself, offering a promising alternative without relying on human or stronger model demonstrations. Self-training, however, requires high-quality model-generated samples, which are hard to obtain for challenging language agent tasks. To address this, we present Reflection-Reinforced Self-Training (Re-ReST), which uses a reflector to refine low-quality generated samples during self-training. The reflector takes the agent’s output and feedback from an external environment (e.g., unit test results in code generation) to produce improved samples. This technique enhances the quality of inferior samples and efficiently enriches the self-training dataset with higher-quality samples. We conduct extensive experiments on open-source language agents across tasks, including multi-hop question answering, sequential decision-making, code generation, visual question answering, and text-to-image generation. The results demonstrate the effectiveness of self-training and Re-ReST in language agent tasks, with self-training improving baselines by 7.6% on HotpotQA and 28.4% on AlfWorld, and Re-ReST further boosting performance by 2.0% and 14.1%, respectively. Our studies also confirm the efficiency of using a reflector to generate high-quality samples for self-training. Moreover, we demonstrate a method to employ reflection during inference without ground-truth feedback, addressing the limitation of previous reflection work.

Problem

• Reducing reliance on costly human or model-generated trajectories
• Improving quality of self-training samples via reflection
• Enhancing language agent performance across diverse tasks

Innovation

• Self-training generates supervision from agent itself
• Reflector refines low-quality samples with feedback
• Reflection during inference without ground-truth feedback