When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon

2026-06-29

AI Summary

This work investigates when online imitation learning outperforms offline methods in the post-training of large language models, focusing on whether its advantage stems from error accumulation or realizability conditions. Through theoretical analysis, information-theoretic tools, and empirical experiments, we propose a novel perspective centered on realizability: under realizable settings, offline imitation learning can already recover expert-level performance; under non-realizable settings, offline methods are fundamentally limited by an information bottleneck, whereas online imitation learning can still effectively approximate the expert policy provided a specific structural misspecification condition holds. Experiments validate the critical role of this condition, revealing the joint influence of policy expressivity, distributional shift, and reward structure on imitation performance.

Abstract

Online imitation learning (IL), particularly on-policy distillation, has emerged as a strong LLM post-training approach, often outperforming offline supervised fine-tuning (SFT). Yet a principled understanding of when and why online interaction helps remains unclear. In this work, we challenge the view that error accumulation is the main source of online IL's advantage, and instead show that the benefits of online interaction depend critically on whether the setting is realizable, i.e., whether the student policy class can represent the expert policy. Under realizability, we empirically find that offline IL already matches expert performance. In contrast, in non-realizable (misspecified) settings, we prove that offline IL encounters an information-theoretic bottleneck even when horizon $H=1$, and propose a structural characterization of misspecification relative to the reward, under which online IL provably achieves high performance despite a large distributional mismatch between the expert and student policies.

Problem

• online imitation learning
• realizability
• non-realizable setting
• LLM post-training
• distributional mismatch

Innovation

• online imitation learning
• realizability
• policy misspecification
• on-policy distillation
• information-theoretic bottleneck