TeaMs-RL: Teaching LLMs to Generate Better Instruction Datasets via Reinforcement Learning

2024-03-13

AI Summary

Existing instruction-tuning methods for LLMs heavily rely on costly human annotations or frequent external LLM queries (e.g., RLHF, self-instruct), leading to high computational expense, privacy risks, and cumbersome pipelines. To address this, we propose TeaMs-RL—the first reinforcement learning framework that directly generates high-quality, diverse instruction datasets. Its core innovation lies in shifting the RL objective from policy optimization to *data generation*: it employs an interpretable, lightweight set of text-editing operations (e.g., rewriting, reordering, constraint injection) to ensure instruction diversity, and introduces a reward-free, self-consistency–based evaluation mechanism for quality assessment. Experiments show that TeaMs-RL reduces external LLM queries by 94.3% (requiring only 5.73% of baseline queries), significantly improves complex instruction understanding and generation, and enables end-to-end, single-stage fine-tuning—fully eliminating the need for subsequent RLHF.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The development of Large Language Models (LLMs) often confronts challenges stemming from the heavy reliance on human annotators in the reinforcement learning with human feedback (RLHF) framework, or the frequent and costly external queries tied to the self-instruct paradigm. In this work, we pivot to Reinforcement Learning (RL) -- but with a twist. Diverging from the typical RLHF, which refines LLMs following instruction data training, we use RL to directly generate the foundational instruction dataset that alone suffices for fine-tuning. Our method, TeaMs-RL, uses a suite of textual operations and rules, prioritizing the diversification of training datasets. It facilitates the generation of high-quality data without excessive reliance on external advanced models, paving the way for a single fine-tuning step and negating the need for subsequent RLHF stages. Our findings highlight key advantages of our approach: reduced need for human involvement and fewer model queries (only 5.73% of the strong baseline's total), along with enhanced capabilities of LLMs in crafting and comprehending complex instructions compared to strong baselines, and substantially improved model privacy protection. Code is available at the link: https://github.com/SafeRL-Lab/TeaMs-RL

Problem

• Reduces reliance on human annotators in LLM training
• Minimizes costly external queries for instruction datasets
• Enhances LLM capabilities and privacy protection

Innovation

• Reinforcement Learning for dataset generation
• Reduces human involvement and model queries
• Enhances LLM capabilities and privacy protection