A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)

2026-02-16

AI Summary

This work addresses the lack of systematic guidance in efficiently selecting the most valuable subset of instructions for instruction tuning of large language models. The authors propose a unified framework that formalizes existing selection algorithms as minimizing an approximation distance between a query set and the chosen subset, and derive a novel generalization bound. Through controlled experiments across models, tasks, and budget levels—complemented by theoretical analysis—they systematically disentangle the effects of data representations and selection algorithms. Their findings reveal that gradient embeddings are the only representation consistently predictive of downstream performance, achieving optimal results when combined with greedy round-robin selection under low budgets; however, this advantage diminishes as the budget increases.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Online Learning & Bandits
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the target task. Despite growing interest, the literature on targeted instruction selection remains fragmented and opaque: methods vary widely in selection budgets, often omit zero-shot baselines, and frequently entangle the contributions of key components. As a result, practitioners lack actionable guidance on selecting instructions for their target tasks. In this work, we aim to bring clarity to this landscape by disentangling and systematically analyzing the two core ingredients: data representation and selection algorithms. Our framework enables controlled comparisons across models, tasks, and budgets. We find that only gradient-based data representations choose subsets whose similarity to the query consistently predicts performance across datasets and models. While no single method dominates, gradient-based representations paired with a greedy round-robin selection algorithm tend to perform best on average at low budgets, but these benefits diminish at larger budgets. Finally, we unify several existing selection algorithms as forms of approximate distance minimization between the selected subset and the query set, and support this view with new generalization bounds. More broadly, our findings provide critical insights and a foundation for more principled data selection in LLM fine-tuning. The code is available at https://github.com/dcml-lab/targeted-instruction-selection.

Problem

• instruction fine-tuning
• targeted instruction selection
• data selection
• large language models
• query-based selection

Innovation

• instruction selection
• data representation
• selection algorithm
• gradient-based representation
• distance minimization