RAISE: Reinforenced Adaptive Instruction Selection For Large Language Models

2025-04-09

AI Summary

Static, heuristic data selection in instruction tuning of large language models (LLMs) leads to suboptimal training efficiency. Method: This paper proposes the first dynamic framework that fully embeds instruction selection into the fine-tuning loop, formulating it as a task-objective-driven sequential decision-making process via Proximal Policy Optimization (PPO). At each training step, the framework dynamically evaluates instruction quality and gradient impact to adaptively select samples with maximal expected performance gain. Contribution/Results: Unlike fixed filtering strategies, our approach offers strong interpretability and task-specific optimization. Experiments demonstrate that updating only 1% of training steps suffices to surpass full-dataset fine-tuning and all baseline selection methods, achieving significantly faster convergence and higher final performance across diverse instruction-following benchmarks.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In the instruction fine-tuning of large language models (LLMs), it has become a consensus that a few high-quality instructions are superior to a large number of low-quality instructions. At present, many instruction selection methods have been proposed, but most of these methods select instruction based on heuristic quality metrics, and only consider data selection before training. These designs lead to insufficient optimization of instruction fine-tuning, and fixed heuristic indicators are often difficult to optimize for specific tasks. So we designed a dynamic, task-objective-driven instruction selection framework RAISE(Reinforenced Adaptive Instruction SElection), which incorporates the entire instruction fine-tuning process into optimization, selecting instruction at each step based on the expected impact of instruction on model performance improvement. Our approach is well interpretable and has strong task-specific optimization capabilities. By modeling dynamic instruction selection as a sequential decision-making process, we use RL to train our selection strategy. Extensive experiments and result analysis prove the superiority of our method compared with other instruction selection methods. Notably, RAISE achieves superior performance by updating only 1% of the training steps compared to full-data training, demonstrating its efficiency and effectiveness.

Problem

• Optimizing instruction selection for LLM fine-tuning
• Dynamic task-driven instruction selection framework
• Improving model performance with minimal training steps

Innovation

• Dynamic task-objective-driven instruction selection framework
• RL-based sequential decision-making for instruction selection
• Optimizes instruction fine-tuning with minimal training steps