The Pitfalls of Imitation Learning when Actions are Continuous

2025-03-12

AI Summary

This paper identifies a fundamental error-amplification problem in imitation learning for continuous state-action spaces: even under stable system dynamics and smooth, deterministic expert policies, any smooth deterministic imitator incurs execution errors that grow exponentially with task horizon—a theoretical bottleneck pervasive in behavioral cloning and offline reinforcement learning. The authors provide the first rigorous proof of this exponential amplification in continuous-action settings, and identify viable mitigation strategies: adopting nonsmooth, non-Markovian, or highly stochastic policies, or leveraging expert datasets with sufficient action-space dispersion. Leveraging contraction theory and control-theoretic analysis, the paper further demonstrates that parametrization techniques—such as action chunking and diffusion-based policy representations—significantly suppress error accumulation. These results establish critical theoretical limits for robotic imitation learning and yield novel design principles for robust policy learning in continuous domains.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Behavior Learning & Control
• Multiagent Systems: Multiagent Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Social networks and social learning
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We study the problem of imitating an expert demonstrator in a discrete-time, continuous state-and-action control system. We show that, even if the dynamics are stable (i.e. contracting exponentially quickly), and the expert is smooth and deterministic, any smooth, deterministic imitator policy necessarily suffers error on execution that is exponentially larger, as a function of problem horizon, than the error under the distribution of expert training data. Our negative result applies to both behavior cloning and offline-RL algorithms, unless they produce highly"improper"imitator policies--those which are non-smooth, non-Markovian, or which exhibit highly state-dependent stochasticity--or unless the expert trajectory distribution is sufficiently"spread."We provide experimental evidence of the benefits of these more complex policy parameterizations, explicating the benefits of today's popular policy parameterizations in robot learning (e.g. action-chunking and Diffusion Policies). We also establish a host of complementary negative and positive results for imitation in control systems.

Problem

• Exponential error growth in continuous action imitation learning
• Limitations of smooth deterministic policies in expert imitation
• Benefits of complex policy parameterizations in robot learning

Innovation

• Examines imitation learning in continuous control systems
• Highlights limitations of smooth deterministic policies
• Advocates complex policies like action-chunking, Diffusion Policies