Stronger Models are NOT Stronger Teachers for Instruction Tuning

2024-11-11

arXiv.org

AI Summary

This work challenges the implicit assumption in instruction tuning that “larger or stronger models necessarily make better teachers,” identifying and naming this phenomenon the *Large Model Paradox*. Through systematic evaluation of 20 response generators (teachers) across 5 base models, we find no monotonic positive correlation between teacher capability and student performance. To address this, we propose **Compatibility-Aware Reward (CAR)**—the first quantitative metric explicitly modeling teacher–base model compatibility, moving beyond conventional unidirectional quality-based evaluation (e.g., relying solely on teacher output quality). Via multi-model ablation, reward modeling, and empirical analysis, we demonstrate that CAR significantly outperforms existing metrics (e.g., ROUGE, BERTScore) in predicting teacher effectiveness and improving downstream instruction-following performance.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Instruction tuning has been widely adopted to ensure large language models (LLMs) follow user instructions effectively. The resulting instruction-following capabilities of LLMs heavily rely on the instruction datasets used for tuning. Recently, synthetic instruction datasets have emerged as an economically viable solution to provide LLMs diverse and high-quality instructions. However, existing approaches typically assume that larger or stronger models are stronger teachers for instruction tuning, and hence simply adopt these models as response generators to the synthetic instructions. In this paper, we challenge this commonly-adopted assumption. Our extensive experiments across five base models and twenty response generators reveal that larger and stronger models are not necessarily stronger teachers of smaller models. We refer to this phenomenon as the Larger Models' Paradox. We observe that existing metrics cannot precisely predict the effectiveness of response generators since they ignore the compatibility between teachers and base models being fine-tuned. We thus develop a novel metric, named as Compatibility-Adjusted Reward (CAR) to measure the effectiveness of response generators. Our experiments across five base models demonstrate that CAR outperforms almost all baselines.

Problem

• Challenges assumption stronger models teach better
• Introduces Compatibility-Adjusted Reward for effectiveness
• Explores Larger Models' Paradox in teaching

Innovation

• Challenges larger models as stronger teachers
• Introduces Compatibility-Adjusted Reward (CAR) metric
• Demonstrates Larger Models' Paradox in instruction tuning