Language-Critique Imitation Learning from Suboptimal Demonstrations

2026-07-01

AI Summary

This work addresses a key limitation of existing imitation learning methods, which rely on scalar supervision signals and struggle to convey task progress, failure causes, or specific corrective guidance when learning from suboptimal demonstrations. To overcome this, the paper introduces natural language as a fine-grained, structured feedback mechanism, generating descriptive linguistic labels for suboptimal trajectories and proposing a novel language critique loss to directly optimize the policy. The approach is instantiated in two frameworks—Language Critique Behavioral Cloning (LC-BC) and Language Critique Diffusion Policy (LC-DP)—and comes with theoretical performance upper-bound guarantees. Experiments across continuous control tasks in navigation, manipulation, and gaming demonstrate that the proposed method significantly outperforms current imitation learning and offline reinforcement learning baselines, confirming the effectiveness and robustness of language-based supervision.

Abstract

Prior work on imitation learning from suboptimal demonstrations typically relies on compressed supervision signals such as confidence estimates, discriminator scores, or importance weights. These scalar signals are inherently limited, as they cannot explicitly express intermediate reasoning about task progress, failure modes, or corrective actions. We propose a language-critique framework for imitation learning from suboptimal demonstrations that instead leverages natural language as a structured supervision signal, avoiding the collapse of expressive feedback into scalars. Our method first constructs language labels from demonstrations that explicitly describe current progress, identify suboptimal behaviors, and provide fine-grained corrective guidance. We then introduce a language-critique loss that directly trains policies using these structured signals without reducing them to scalars, and instantiate it for both behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We further provide a theoretical result showing that the proposed objective upper-bounds the expert performance gap under standard assumptions. Empirically, we evaluate on diverse continuous control tasks spanning navigation, manipulation, and gameplay, where our methods consistently outperform strong imitation learning and offline reinforcement learning baselines. These results demonstrate that language can serve as a powerful and structured form of supervision for learning robust policies from suboptimal data.

Problem

• imitation learning
• suboptimal demonstrations
• language supervision
• structured feedback
• policy learning

Innovation

• language-critique
• imitation learning
• suboptimal demonstrations
• structured supervision
• natural language feedback