SelectIT: Selective Instruction Tuning for LLMs via Uncertainty-Aware Self-Reflection

2024-02-26

AI Summary

High-quality data selection for instruction tuning (IT) is costly, often requiring external models or human annotations. Method: We propose SelectIT, an intrinsic uncertainty-aware self-reflection data selection mechanism leveraging large language models’ (LLMs) own output entropy and confidence estimates—without auxiliary models, labels, or additional training. Through self-reflective prompting and dynamic quality assessment, SelectIT enables zero-cost, iterative refinement of instruction data. Contribution/Results: We introduce Selective Alpaca—the first high-quality subset (10% of Alpaca-GPT4) optimized via SelectIT—which consistently improves instruction-following performance across diverse base models (Llama-2/3, Qwen) and cross-domain tasks, even surpassing full-data fine-tuning baselines. This work establishes the first LLM uncertainty-driven, self-supervised data selection paradigm, offering a computationally efficient and cost-effective pathway to instruction alignment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Instruction tuning (IT) is crucial to tailoring large language models (LLMs) towards human-centric interactions. Recent advancements have shown that the careful selection of a small, high-quality subset of IT data can significantly enhance the performance of LLMs. Despite this, common approaches often rely on additional models or data, which increases costs and limits widespread adoption. In this work, we propose a novel approach, termed SelectIT, that capitalizes on the foundational capabilities of the LLM itself. Specifically, we exploit the intrinsic uncertainty present in LLMs to more effectively select high-quality IT data, without the need for extra resources. Furthermore, we introduce a curated IT dataset, the Selective Alpaca, created by applying SelectIT to the Alpaca-GPT4 dataset. Empirical results demonstrate that IT using Selective Alpaca leads to substantial model ability enhancement. The robustness of SelectIT has also been corroborated in various foundation models and domain-specific tasks. Our findings suggest that longer and more computationally intensive IT data may serve as superior sources of IT, offering valuable insights for future research in this area. Data, code, and scripts are freely available at https://github.com/Blue-Raincoat/SelectIT.

Problem

• Large Language Models
• Uncertainty Calibration
• Efficient Instruction Tuning

Innovation

• SelectIT
• Self-inspection Uncertainty
• Selective Refinement