Behavior Cloning is Not All You Need: The Optimality of On-Policy Distillation for Noisy Expert Feedback

2026-06-29

AI Summary

This work addresses the challenge of imitation learning when only noisy expert demonstrations are available. While offline imitation learning struggles to match the performance of a clean expert under such conditions, online policy distillation (OPD) empirically outperforms it, yet lacks theoretical justification. The paper introduces the first formal model of noisy experts and theoretically establishes that OPD achieves polynomial sample complexity in the task horizon, whereas offline methods require exponentially many samples. It further identifies necessary conditions to eliminate dependence on task length, proposes a novel OPD algorithm, derives fundamental lower bounds on sample complexity, and extends the framework to settings with unknown noise. These results provide a theoretical foundation for OPD’s superiority over supervised fine-tuning, yield an alternative loss function suitable for language model training, and guarantee effective learning even with unknown noise when the clean expert is deterministic.

Abstract

Imitation Learning is a natural framework for learning in sequential decision-making systems and has emerged as the dominant paradigm through which we understand language model training. A central puzzle is that, while in theory offline IL can be horizon-free and optimal, in practice online methods such as on-policy distillation often outperform offline methods such as supervised fine-tuning. We propose a noisy expert model to explain this gap, in which the learner only has access to a noisy version of the expert's policy, but wishes to compete against the reward achieved by a clean expert, motivated by the fact that in many applications, e.g. training language models to perform long chains of thought, the expert is often imperfect. In this setting, we show a sharp separation between offline and online IL. Offline learning from noisy trajectories is fundamentally hard: to compete with the clean expert, the sample complexity must grow exponentially, in contradistinction to the clean expert setting where no explicit horizon dependence exists. In contrast, we prove that online interaction with the noisy expert via a novel variant of OPD enables polynomial dependence on the horizon in general. We further show that, under a natural condition on the expert noise distribution, which we show to be necessary for any horizon-free sample complexity, one can obtain such a guarantee, although our proposed algorithm sacrifices statistical efficiency in its dependence on the size of the policy class. Our analysis leads to an alternative loss function that is commonly considered empirically for LM training. We further provide algorithms and lower bounds, and extend our results to the more realistic setting of unknown corruption when the clean expert is deterministic, thereby providing a theoretical foundation for why OPD can outperform SFT when training language models from imperfect teachers.

Problem

• Imitation Learning
• Noisy Expert
• On-Policy Distillation
• Offline Learning
• Sample Complexity

Innovation

• Imitation Learning
• On-Policy Distillation
• Noisy Expert
• Sample Complexity
• Behavior Cloning