🤖 AI Summary
Existing knowledge distillation methods for in-context learning (ICL) with small language models focus solely on aligning student outputs with teacher predictions, neglecting the teacher’s input example preferences—thereby limiting generalization. To address this, we propose BiAlign, the first framework that jointly models and optimizes student imitation of teacher behavior along *both* input example preference and output distribution dimensions. Methodologically, BiAlign integrates token-level probabilistic distillation with a novel ranking loss to explicitly capture differential sensitivity to demonstration examples. Evaluated across language understanding, reasoning, and code generation tasks, BiAlign consistently outperforms state-of-the-art distillation approaches—achieving superior robustness and ICL transfer efficiency with fewer parameters and lower computational overhead. This work establishes a new paradigm for enhancing the ICL capability of compact models through bidirectional behavioral alignment.
📝 Abstract
Large language models (LLMs) have shown impressive few-shot generalization on many tasks via in-context learning (ICL). Despite their success in showing such emergent abilities, the scale and complexity of larger models also lead to unprecedentedly high computational demands and deployment challenges. In reaction, researchers explore transferring the powerful capabilities of larger models to more efficient and compact models by typically aligning the output of smaller (student) models with that of larger (teacher) models. Existing methods either train student models on the generated outputs of teacher models or imitate their token-level probability distributions. However, these distillation methods pay little to no attention to the input, which also plays a crucial role in ICL. Based on the finding that the performance of ICL is highly sensitive to the selection of demonstration examples, we propose Bidirectional Alignment (BiAlign) to fully leverage the models' preferences for ICL examples to improve the ICL abilities of student models. Specifically, we introduce the alignment of input preferences between student and teacher models by incorporating a novel ranking loss, in addition to aligning the token-level output distribution. With extensive experiments and analysis, we demonstrate that BiAlign can consistently outperform existing baselines on a variety of tasks involving language understanding, reasoning, and coding.
