Sparse Rewards Can Self-Train Dialogue Agents

2024-09-06

arXiv.org

AI Summary

Large language model (LLM)-based dialogue agents struggle with scalability and cost in multi-turn task settings due to their reliance on high-quality human feedback for reinforcement learning. Method: This paper proposes a human-feedback-free self-training paradigm, built upon a sparse-reward simulated environment (ToolWOZ) and a novel self-alignment algorithm—JOSH—that enables iterative optimization using only self-generated supervision signals. JOSH is the first method to support fully autonomous, continuous improvement of LLM dialogue agents solely from sparse environmental rewards. Results: Experiments demonstrate substantial gains in tool-use capability across both compact and state-of-the-art LLMs, achieving significant performance improvements on multiple benchmarks—including ToolAlpaca, APIBench, and ToolBench—while preserving general language competence with no degradation in open-ended generation or reasoning abilities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advancements in state-of-the-art (SOTA) Large Language Model (LLM) agents, especially in multi-turn dialogue tasks, have been primarily driven by supervised fine-tuning and high-quality human feedback. However, as base LLM models continue to improve, acquiring meaningful human feedback has become increasingly challenging and costly. In certain domains, base LLM agents may eventually exceed human capabilities, making traditional feedback-driven methods impractical. In this paper, we introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback. Our method, Juxtaposed Outcomes for Simulation Harvesting (JOSH), is a self-alignment algorithm that leverages a sparse reward simulation environment to extract ideal behaviors and further train the LLM on its own outputs. We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ. We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks. Our code and data are publicly available on GitHub at https://github.com/asappresearch/josh-llm-simulation-training

Problem

• Enhancing LLM agents without human feedback
• Self-training using sparse reward simulations
• Improving tool-based dialogue interactions autonomously

Innovation

• Self-alignment algorithm with sparse rewards
• Simulation environment for autonomous training
• Preserves general capabilities while improving performance